Compare commits

..

105 Commits

Author SHA1 Message Date
刘祥超
fa6be81abe 特殊页面可以直接使用HTML 2021-10-10 10:35:05 +08:00
刘祥超
1b2f01f0f4 把tcp/udp的连接数记为访问量,增加tcp域名排名记录(需要SNI连接) 2021-10-08 15:50:43 +08:00
刘祥超
09467a4d08 WAF模式从pass改为bypass 2021-10-07 13:51:26 +08:00
刘祥超
a17bbc3df1 缓存预热判断请求来源的时候增加IPv6回路地址判断 2021-10-06 16:35:39 +08:00
刘祥超
dc495c70b3 服务支持自定义访客IP地址获取方式/对X-Real-IP等Header值进行有效性验证 2021-10-06 11:40:48 +08:00
刘祥超
d0cf145f85 优化WAF动作排序 2021-10-06 09:39:57 +08:00
刘祥超
95f1e61489 WAF动作block和record_ip同时存在时,优先执行record_ip 2021-10-06 08:56:38 +08:00
刘祥超
42a0161312 WAF模式为defend时IP黑名单才生效 2021-10-04 18:22:51 +08:00
刘祥超
729443f0b4 大幅优化IP名单查询速度 2021-10-04 17:42:38 +08:00
刘祥超
3888565c0f 根据系统内存自动调节ttlcache的最大条目 2021-10-04 09:12:17 +08:00
刘祥超
1ca967534a 开启缓存后覆盖源站的ETag和Last-Modified 2021-10-04 08:41:44 +08:00
刘祥超
c01bb57dea 不把499状态码加入状态码统计 2021-10-04 08:41:13 +08:00
刘祥超
adadb52d4e 优化WebP+缓存 2021-10-03 18:00:57 +08:00
刘祥超
38a7cc17da 修复WAF策略模式为空导致动作不起作用的问题 2021-10-03 08:35:28 +08:00
刘祥超
246bb45614 限制WebP转换时消耗的内存总量 2021-10-01 18:59:44 +08:00
刘祥超
b320d2dc58 修复WebP缓存长度可能不正确的问题 2021-10-01 17:20:37 +08:00
刘祥超
96c63300f4 恢复源站默认连接数限制 2021-10-01 16:45:46 +08:00
刘祥超
3eaf090aac 缓存内容也支持压缩 2021-10-01 16:24:29 +08:00
刘祥超
b157448ad2 支持自动转换图像文件为WebP 2021-10-01 16:24:17 +08:00
刘祥超
44998a23fb 反向代理去除默认跟源站的连接数限制 2021-10-01 11:17:43 +08:00
刘祥超
e3e30ffee5 节点启动时如果加载的是本地配置则在网络恢复后重新加载配置 2021-10-01 11:13:36 +08:00
刘祥超
12bddc6e82 WAF策略增加观察模式和通过模式 2021-09-30 11:30:58 +08:00
刘祥超
771d2d8013 支持brotli和deflate压缩 2021-09-29 19:37:07 +08:00
刘祥超
3bf94bc032 优化WAF关闭连接操作 2021-09-29 11:06:00 +08:00
刘祥超
a1aa2b9224 Block动作增加默认时间60秒 2021-09-29 09:19:45 +08:00
刘祥超
8d28ba3426 缓存条件增加最小内容尺寸配置 2021-09-26 15:01:46 +08:00
刘祥超
5a72c10d83 版本改为0.3.2 2021-09-26 10:09:56 +08:00
刘祥超
ec113c59ab 将版本修改为0.3.1 2021-09-23 21:00:06 +08:00
刘祥超
7f9d95ba37 修复当缓存内容为空时无法响应缓存的Bug 2021-09-22 21:13:31 +08:00
刘祥超
558314265a 修改部分命名 2021-09-22 19:40:11 +08:00
刘祥超
c793dd9d8c 特殊页面中的URL抓取的内容也支持请求变量 2021-09-21 10:13:30 +08:00
刘祥超
33e899a008 特殊页面支持请求变量 2021-09-21 09:29:17 +08:00
刘祥超
903f511139 反向代理源站实现使用域名分组 2021-09-20 11:54:37 +08:00
刘祥超
49dafafdc5 修复反向代理Hash调度算法无法生效的Bug 2021-09-20 09:59:24 +08:00
刘祥超
c6b01dc10a 修复反向代理sticky调度算法无法生效的Bug 2021-09-20 09:55:42 +08:00
刘祥超
1119b351aa 内存缓存增加最大数量限制 2021-09-19 16:11:46 +08:00
刘祥超
7b8ef8e85b 配置加载成功后才启动某些任务 2021-09-16 15:58:10 +08:00
刘祥超
dd41d88647 指标数据队列增加最大数量限制,防止过载 2021-09-03 15:38:56 +08:00
刘祥超
699cea4382 请求源站错误时增加503、504错误 2021-09-01 08:48:03 +08:00
刘祥超
dfb0e60acc 缓存文件列表关闭时也清除内存缓存 2021-08-29 08:59:32 +08:00
刘祥超
417e10970a 缓存预热时不重复写入 2021-08-26 15:48:09 +08:00
刘祥超
6437875979 优化ACME读取速度 2021-08-25 17:32:53 +08:00
刘祥超
beb251a3cc 调整ACME证书申请链接的优先级为最高,避免因URL跳转而导致无法申请证书 2021-08-25 16:50:07 +08:00
刘祥超
8069f20c77 修改注释 2021-08-25 16:46:05 +08:00
刘祥超
8161270f47 优化指标统计写入数据逻辑 2021-08-22 10:07:04 +08:00
刘祥超
be7de223af 提升文件缓存读取效率大约5% 2021-08-21 21:44:34 +08:00
刘祥超
7469f528a8 通过内存缓存提升文件缓存效率大约20% 2021-08-21 21:06:48 +08:00
刘祥超
1f88db8829 阶段性提交 2021-08-21 20:45:11 +08:00
刘祥超
844a5fc321 调整版本为0.3.0 2021-08-17 09:44:37 +08:00
刘祥超
06db07ac4b 优化代码 2021-08-07 16:27:42 +08:00
刘祥超
00df3fca84 调整版本为0.2.9 2021-08-07 16:11:48 +08:00
刘祥超
4d679e31bc 改进WAF record_ip动作 2021-08-04 15:35:22 +08:00
刘祥超
06e27bb469 调整版本 2021-08-04 15:34:55 +08:00
刘祥超
684ba7082b 修复统计指标数据上传不完整的问题 2021-08-03 14:02:15 +08:00
刘祥超
8934962de2 调整版本号 2021-08-03 10:40:46 +08:00
刘祥超
0cf37f25dc 优化源站调度 2021-08-02 16:14:56 +08:00
刘祥超
d7a6d71fea 优化源站调度 2021-08-01 21:56:02 +08:00
刘祥超
a26f7941d5 WAF策略和缓存策略跟随集群 2021-08-01 14:54:06 +08:00
刘祥超
f5365e5420 修复IPv6访问可能导致进程异常退出的Bug 2021-08-01 09:20:07 +08:00
刘祥超
56d21f867b 增加referer.host请求变量 2021-07-26 15:37:47 +08:00
刘祥超
d18a301c61 WAF get302和post307只有在HTTP/1的情况下才在跳转前关闭连接 2021-07-26 14:33:06 +08:00
刘祥超
afb937030c 自动跳转到HTTPS可以设置允许和排除的域名 2021-07-26 11:23:57 +08:00
刘祥超
8faa82c453 域名调整为0.2.6 2021-07-26 11:23:52 +08:00
刘祥超
b17b63aec5 删除一个不需要的文件 2021-07-25 17:40:31 +08:00
刘祥超
c30dbb811f 优化Daemon代码 2021-07-25 17:39:09 +08:00
刘祥超
a58816361e 使用Sock管理进程启停 2021-07-25 17:14:44 +08:00
刘祥超
8d37aefd95 更新编译脚本 2021-07-25 16:28:36 +08:00
刘祥超
df7fee966e 缓存路径为/时,不再提示错误 2021-07-21 11:55:08 +08:00
刘祥超
01cfccebbd 缓存结束后增加Content-Length对比 2021-07-20 19:48:08 +08:00
刘祥超
6bd7da5e6e build脚本增加arm编译 2021-07-20 19:01:49 +08:00
刘祥超
dcba9c2f3e 调整格式等 2021-07-20 19:01:37 +08:00
刘祥超
f38e80e82d 自动替换API节点时增加对新节点的测试 2021-07-20 18:17:25 +08:00
刘祥超
9bd38094c3 将WAF模板中的cc修改为cc2 2021-07-19 11:01:38 +08:00
刘祥超
7e37fc3b80 实现新的CC 2021-07-19 10:49:56 +08:00
刘祥超
d775dfeeaa WAF增加多个动作 2021-07-18 15:51:49 +08:00
刘祥超
0486f86898 增加攻击拦截统计 2021-07-13 11:04:38 +08:00
刘祥超
102157c893 节点看板增加缓存目录用量 2021-07-08 19:43:30 +08:00
刘祥超
dbd92368ae 文件缓存统计去除过期时间条件 2021-07-08 09:19:41 +08:00
刘祥超
9e418e73bf 节点状态上报流量 2021-07-07 15:18:01 +08:00
刘祥超
5d40eec163 实现节点看板(仅对企业版开放) 2021-07-06 20:06:57 +08:00
刘祥超
1a7a67238d 增加域名统计 2021-07-05 11:36:50 +08:00
刘祥超
6707437bae 指标数据增加总和数据 2021-07-01 10:39:56 +08:00
刘祥超
df7859387d 实现基础的统计指标 2021-06-30 20:01:00 +08:00
刘祥超
889c52330d 修改版本为0.2.5 2021-06-28 10:32:18 +08:00
刘祥超
0e912b79cd TOA通讯失败时,关闭连接 2021-06-27 17:31:10 +08:00
刘祥超
12f3916e45 ip2region增加IP格式检查 2021-06-27 17:30:45 +08:00
刘祥超
7813e2c3d2 更新TOA 2021-06-24 17:38:29 +08:00
刘祥超
54eff9bfae 删除TOA 2021-06-24 17:29:40 +08:00
刘祥超
635cdd4338 上传TOA编译文件 2021-06-24 16:59:52 +08:00
刘祥超
4c64d3ab0f 实现公用的IP名单 2021-06-23 13:14:37 +08:00
刘祥超
93a5c90fcb 应用网站自定义的WAF出站规则 2021-06-21 15:29:07 +08:00
刘祥超
eb5e863146 变更版本 2021-06-21 14:43:29 +08:00
刘祥超
78e566174f 更新部分Go Package 2021-06-20 19:23:35 +08:00
刘祥超
dd93a93ba9 访问控制支持基本认证和子请求认证 2021-06-19 21:35:57 +08:00
刘祥超
aaa6899976 优化文件缓存 2021-06-17 21:13:21 +08:00
刘祥超
e04e3287b4 调整版本为0.2.3 2021-06-17 18:45:59 +08:00
刘祥超
e715693156 调整版本为0.2.2 2021-06-17 18:05:35 +08:00
刘祥超
2798c3c5e5 修复移除内存缓存死锁的Bug 2021-06-17 18:04:56 +08:00
刘祥超
489e081720 修复可能导致死锁冲突的Bug 2021-06-16 16:10:02 +08:00
刘祥超
77a8eb5c1a 触发浏览器304缓存也算缓存命中 2021-06-16 11:04:19 +08:00
刘祥超
3b7d2b91c7 调整版本为0.2.1 2021-06-16 08:30:26 +08:00
刘祥超
20b299fb3b 优化错误提示 2021-06-16 08:29:38 +08:00
刘祥超
e4b3d2b2aa 不提示一些客户端错误 2021-06-16 07:52:06 +08:00
刘祥超
d237ee6b5b 节点启动错误时自动尝试从本地读取缓存数据 2021-06-15 10:55:49 +08:00
刘祥超
34aa6125df 修改一处注释错别字 2021-06-15 10:47:40 +08:00
214 changed files with 6577 additions and 2059 deletions

2
.gitignore vendored Normal file
View File

@@ -0,0 +1,2 @@
*_plus.go
*-plus.sh

View File

@@ -7,6 +7,7 @@ function build() {
DIST=$ROOT/"../dist/${NAME}"
OS=${1}
ARCH=${2}
TAG=${3}
if [ -z $OS ]; then
echo "usage: build.sh OS ARCH"
@@ -16,6 +17,9 @@ function build() {
echo "usage: build.sh OS ARCH"
exit
fi
if [ -z $TAG ]; then
TAG="community"
fi
echo "checking ..."
ZIP_PATH=$(which zip)
@@ -24,8 +28,8 @@ function build() {
exit
fi
echo "building v${VERSION}/${OS}/${ARCH} ..."
ZIP="${NAME}-${OS}-${ARCH}-v${VERSION}.zip"
echo "building v${VERSION}/${OS}/${ARCH}/${TAG} ..."
ZIP="${NAME}-${OS}-${ARCH}-${TAG}-v${VERSION}.zip"
echo "copying ..."
if [ ! -d $DIST ]; then
@@ -66,6 +70,10 @@ function build() {
CC_PATH="aarch64-linux-musl-gcc"
CXX_PATH="aarch64-linux-musl-g++"
fi
if [ "${ARCH}" == "arm" ]; then
CC_PATH="arm-linux-musleabi-gcc"
CXX_PATH="arm-linux-musleabi-g++"
fi
if [ "${ARCH}" == "mips64" ]; then
CC_PATH="mips64-linux-musl-gcc"
CXX_PATH="mips64-linux-musl-g++"
@@ -76,9 +84,9 @@ function build() {
fi
fi
if [ ! -z $CC_PATH ]; then
env CC=$MUSL_DIR/$CC_PATH CXX=$MUSL_DIR/$CXX_PATH GOOS=${OS} GOARCH=${ARCH} CGO_ENABLED=1 go build -o $DIST/bin/${NAME} -ldflags "-linkmode external -extldflags -static -s -w" $ROOT/../cmd/edge-node/main.go
env CC=$MUSL_DIR/$CC_PATH CXX=$MUSL_DIR/$CXX_PATH GOOS=${OS} GOARCH=${ARCH} CGO_ENABLED=1 go build -tags $TAG -o $DIST/bin/${NAME} -ldflags "-linkmode external -extldflags -static -s -w" $ROOT/../cmd/edge-node/main.go
else
env GOOS=${OS} GOARCH=${ARCH} CGO_ENABLED=1 go build -o $DIST/bin/${NAME} -ldflags="-s -w" $ROOT/../cmd/edge-node/main.go
env GOOS=${OS} GOARCH=${ARCH} CGO_ENABLED=1 go build -tags $TAG -o $DIST/bin/${NAME} -ldflags="-s -w" $ROOT/../cmd/edge-node/main.go
fi
# delete hidden files
@@ -110,4 +118,4 @@ function lookup-version() {
fi
}
build $1 $2
build $1 $2 $3

View File

@@ -1 +1 @@
index.*
*

View File

@@ -1 +0,0 @@
edge-toa

BIN
build/edge-toa/edge-toa Executable file

Binary file not shown.

View File

@@ -9,7 +9,7 @@
<h3>403 Forbidden</h3>
<p>Sorry, your access to the page has been denied. Please try again later.</p>
<footer>Powered by TeaEdge.</footer>
<footer>Powered by GoEdge.</footer>
</body>
</html>

View File

@@ -9,7 +9,7 @@
<h3>404 Not Found</h3>
<p>Sorry, the page you are looking for is not found. Please try again later.</p>
<footer>Powered by TeaEdge.</footer>
<footer>Powered by GoEdge.</footer>
</body>
</html>

View File

@@ -9,7 +9,7 @@
<h3>An error occurred.</h3>
<p>Sorry, the page you are looking for is currently unavailable. Please try again later.</p>
<footer>Powered by TeaEdge.</footer>
<footer>Powered by GoEdge.</footer>
</body>
</html>

View File

@@ -9,7 +9,7 @@
<h3>The website is shutdown.</h3>
<p>Sorry, the page you are looking for is currently unavailable. Please try again later.</p>
<footer>Powered by TeaEdge.</footer>
<footer>Powered by GoEdge.</footer>
</body>
</html>

View File

@@ -9,7 +9,7 @@
<h3>网站升级中</h3>
<p>为了给您提供更好的服务,我们正在升级网站,请稍后重新访问。</p>
<footer>Powered by TeaEdge.</footer>
<footer>Powered by GoEdge.</footer>
</body>
</html>

View File

@@ -9,7 +9,7 @@
<h3>网站暂时关闭</h3>
<p>网站已被暂时关闭,请耐心等待我们的重新开通通知。</p>
<footer>Powered by TeaEdge.</footer>
<footer>Powered by GoEdge.</footer>
</body>
</html>

View File

@@ -5,15 +5,12 @@ import (
"github.com/TeaOSLab/EdgeNode/internal/apps"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/nodes"
"github.com/iwind/TeaGo/Tea"
_ "github.com/iwind/TeaGo/bootstrap"
"github.com/iwind/TeaGo/logs"
"github.com/iwind/TeaGo/types"
"io/ioutil"
"github.com/iwind/gosock/pkg/gosock"
"net/http"
_ "net/http/pprof"
"os"
"syscall"
)
func main() {
@@ -40,25 +37,13 @@ func main() {
fmt.Println("done")
})
app.On("quit", func() {
pidFile := Tea.Root + "/bin/pid"
data, err := ioutil.ReadFile(pidFile)
var sock = gosock.NewTmpSock(teaconst.ProcessName)
_, err := sock.Send(&gosock.Command{Code: "quit"})
if err != nil {
fmt.Println("[ERROR]quit failed: " + err.Error())
return
}
pid := types.Int(string(data))
if pid == 0 {
fmt.Println("[ERROR]quit failed: pid=0")
return
}
process, err := os.FindProcess(pid)
if err != nil {
return
}
if process != nil {
_ = process.Signal(syscall.SIGQUIT)
}
fmt.Println("done")
})
app.On("pprof", func() {
// TODO 自己指定端口

23
go.mod
View File

@@ -7,20 +7,27 @@ replace github.com/TeaOSLab/EdgeCommon => ../EdgeCommon
require (
github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d // indirect
github.com/TeaOSLab/EdgeCommon v0.0.0-00010101000000-000000000000
github.com/andybalholm/brotli v1.0.3
github.com/cespare/xxhash v1.1.0
github.com/chai2010/webp v1.1.0
github.com/dchest/captcha v0.0.0-20200903113550-03f5f0333e1f
github.com/dop251/goja v0.0.0-20210804101310-32956a348b49
github.com/go-ole/go-ole v1.2.4 // indirect
github.com/go-yaml/yaml v2.1.0+incompatible
github.com/golang/protobuf v1.4.2
github.com/iwind/TeaGo v0.0.0-20210411134150-ddf57e240c2f
github.com/golang/protobuf v1.5.2
github.com/iwind/TeaGo v0.0.0-20210628135026-38575a4ab060
github.com/iwind/gofcgi v0.0.0-20210528023741-a92711d45f11
github.com/iwind/gosock v0.0.0-20210722083328-12b2d66abec3
github.com/lionsoul2014/ip2region v2.2.0-release+incompatible
github.com/mattn/go-sqlite3 v1.14.7
github.com/mattn/go-sqlite3 v2.0.3+incompatible
github.com/mssola/user_agent v0.5.2
github.com/shirou/gopsutil v2.20.9+incompatible
golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7
golang.org/x/sys v0.0.0-20200519105757-fe76b779f299
golang.org/x/text v0.3.2
google.golang.org/grpc v1.32.0
github.com/shirou/gopsutil v3.21.5+incompatible
github.com/tklauser/go-sysconf v0.3.6 // indirect
golang.org/x/image v0.0.0-20190802002840-cff245a6509b
golang.org/x/net v0.0.0-20210614182718-04defd469f4e
golang.org/x/sys v0.0.0-20210616094352-59db8d763f22
golang.org/x/text v0.3.6
google.golang.org/genproto v0.0.0-20210617175327-b9e0b3197ced // indirect
google.golang.org/grpc v1.38.0
gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776
)

93
go.sum
View File

@@ -7,23 +7,35 @@ github.com/OneOfOne/xxhash v1.2.2 h1:KMrpdQIwFcEqXDklaen+P1axHaj9BSKzvpUUfnHldSE
github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d h1:G0m3OIz70MZUWq3EgK3CesDbo8upS2Vm9/P3FtgI+Jk=
github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d/go.mod h1:3eOhrUMpNV+6aFIbp5/iudMxNCF27Vw2OZgy4xEx0Fg=
github.com/andybalholm/brotli v1.0.3 h1:fpcw+r1N1h0Poc1F/pHbW40cUm/lMEQslZtCkBQ0UnM=
github.com/andybalholm/brotli v1.0.3/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig=
github.com/benbjohnson/clock v1.0.3/go.mod h1:bGMdMPoPVvcYyt1gHDf4J2KE153Yf9BuiUKYMaxlTDM=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/cespare/xxhash v1.1.0 h1:a6HrQnmkObjyL+Gs60czilIUGqrzKutQD6XZog3p+ko=
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
github.com/cespare/xxhash/v2 v2.1.1 h1:6MnRN8NT7+YBpUIWxHtefFZOKTAPgGjpQSxqLNn0+qY=
github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/chai2010/webp v1.1.0 h1:4Ei0/BRroMF9FaXDG2e4OxwFcuW2vcXd+A6tyqTJUQQ=
github.com/chai2010/webp v1.1.0/go.mod h1:LP12PG5IFmLGHUU26tBiCBKnghxx3toZFwDjOYvd3Ow=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
github.com/cncf/udpa/go v0.0.0-20201120205902-5459f2c99403/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dchest/captcha v0.0.0-20200903113550-03f5f0333e1f h1:q/DpyjJjZs94bziQ7YkBmIlpqbVP7yw179rnzoNVX1M=
github.com/dchest/captcha v0.0.0-20200903113550-03f5f0333e1f/go.mod h1:QGrK8vMWWHQYQ3QU9bw9Y9OPNfxccGzfb41qjvVeXtY=
github.com/dgryski/go-rendezvous v0.0.0-20200624174652-8d2f3be8b2d9/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
github.com/dlclark/regexp2 v1.4.1-0.20201116162257-a2a8dda75c91 h1:Izz0+t1Z5nI16/II7vuEo/nHjodOg0p7+OiDpjX5t1E=
github.com/dlclark/regexp2 v1.4.1-0.20201116162257-a2a8dda75c91/go.mod h1:2pZnwuY/m+8K6iRw6wQdMtk+rH5tNGR1i55kozfMjCc=
github.com/dop251/goja v0.0.0-20210804101310-32956a348b49 h1:CtSi0QlA2Hy+nOh8JAZoiEBLW5pliAiKJ3l1Iq1472I=
github.com/dop251/goja v0.0.0-20210804101310-32956a348b49/go.mod h1:R9ET47fwRVRPZnOGvHxxhuZcbrMCuiqOz3Rlrh4KSnk=
github.com/dop251/goja_nodejs v0.0.0-20210225215109-d91c329300e7/go.mod h1:hn7BA7c8pLvoGndExHudxTDKZ84Pyvv+90pbBjbTz0Y=
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
github.com/envoyproxy/go-control-plane v0.9.9-0.20210217033140-668b12f5399d/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
@@ -31,6 +43,8 @@ github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2
github.com/go-ole/go-ole v1.2.4 h1:nNBDSCOigTSiarFpYE9J/KtEA1IOW4CNeqT9TQDqCxI=
github.com/go-ole/go-ole v1.2.4/go.mod h1:XCwSNxSkXRo4vlyPy93sltvi/qJq0jqQhjqQNIwKuxM=
github.com/go-redis/redis/v8 v8.0.0-beta.7/go.mod h1:FGJAWDWFht1sQ4qxyJHZZbVyvnVcKQN0E3u5/5lRz+g=
github.com/go-sourcemap/sourcemap v2.1.3+incompatible h1:W1iEw64niKVGogNgBN3ePyLFfuisuzeidWPMPWmECqU=
github.com/go-sourcemap/sourcemap v2.1.3+incompatible/go.mod h1:F8jJfvm2KbVjc5NqelyYJmf/v5J0dwNLS2mL4sNA1Jg=
github.com/go-sql-driver/mysql v1.5.0 h1:ozyZYNQW3x3HtqT1jira07DN2PArx2v7/mN66gGcHOs=
github.com/go-sql-driver/mysql v1.5.0/go.mod h1:DCzpHaOWr8IXmIStZouvnhqoel9Qv2LBy8hT2VhHyBg=
github.com/go-yaml/yaml v2.1.0+incompatible h1:RYi2hDdss1u4YE7GwixGzWwVo47T8UQwnTLB6vQiq+o=
@@ -46,34 +60,42 @@ github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrU
github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8=
github.com/golang/protobuf v1.4.2 h1:+Z5KGCizgyZCbGh1KZqA0fcLLkwbsjIzS4aV2v7wJX0=
github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
github.com/golang/protobuf v1.5.2 h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.0 h1:/QaMHBdZ26BB3SSst0Iwl10Epc+xhTquomWX0oZEB6w=
github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.5 h1:Khx7svrCpmxxtHBq5j2mp/xVjsi8hQMfNLvJFAlrGgU=
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
github.com/iwind/TeaGo v0.0.0-20210411134150-ddf57e240c2f h1:r2O8PONj/KiuZjJHVHn7KlCePUIjNtgAmvLfgRafQ8o=
github.com/iwind/TeaGo v0.0.0-20210411134150-ddf57e240c2f/go.mod h1:KU4mS7QNiZ7QWEuDBk1zw0/Q2LrAPZv3tycEFBsuUwc=
github.com/iwind/TeaGo v0.0.0-20210628135026-38575a4ab060 h1:qdLtK4PDXxk2vMKkTWl5Fl9xqYuRCukzWAgJbLHdfOo=
github.com/iwind/TeaGo v0.0.0-20210628135026-38575a4ab060/go.mod h1:KU4mS7QNiZ7QWEuDBk1zw0/Q2LrAPZv3tycEFBsuUwc=
github.com/iwind/gofcgi v0.0.0-20210528023741-a92711d45f11 h1:DaQjoWZhLNxjhIXedVg4/vFEtHkZhK4IjIwsWdyzBLg=
github.com/iwind/gofcgi v0.0.0-20210528023741-a92711d45f11/go.mod h1:JtbX20untAjUVjZs1ZBtq80f5rJWvwtQNRL6EnuYRnY=
github.com/iwind/gosock v0.0.0-20210722083328-12b2d66abec3 h1:aBSonas7vFcgTj9u96/bWGILGv1ZbUSTLiOzcI1ZT6c=
github.com/iwind/gosock v0.0.0-20210722083328-12b2d66abec3/go.mod h1:H5Q7SXwbx3a97ecJkaS2sD77gspzE7HFUafBO0peEyA=
github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pretty v0.2.1 h1:Fmg33tUaq4/8ym9TJN1x7sLJnHVwhP33CNkpYV/7rwI=
github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/lionsoul2014/ip2region v2.2.0-release+incompatible h1:1qp9iks+69h7IGLazAplzS9Ca14HAxuD5c0rbFdPGy4=
github.com/lionsoul2014/ip2region v2.2.0-release+incompatible/go.mod h1:+ZBN7PBoh5gG6/y0ZQ85vJDBe21WnfbRrQQwTfliJJI=
github.com/mattn/go-sqlite3 v1.14.7 h1:fxWBnXkxfM6sRiuH3bqJ4CfzZojMOLVc0UTsTglEghA=
github.com/mattn/go-sqlite3 v1.14.7/go.mod h1:NyWgC/yNuGj7Q9rpYnZvas74GogHl5/Z4A/KQRfk6bU=
github.com/mattn/go-sqlite3 v2.0.3+incompatible h1:gXHsfypPkaMZrKbD5209QV9jbUTJKjyR5WD3HYQSd+U=
github.com/mattn/go-sqlite3 v2.0.3+incompatible/go.mod h1:FPy6KqzDD04eiIsT53CuJW3U88zkxoIYsOqkbpncsNc=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/mssola/user_agent v0.5.2 h1:CZkTUahjL1+OcZ5zv3kZr8QiJ8jy2H08vZIEkBeRbxo=
github.com/mssola/user_agent v0.5.2/go.mod h1:TTPno8LPY3wAIEKRpAtkdMT0f8SE24pLRGPahjCH4uw=
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e h1:fD57ERR4JtEqsWbfPhv4DMiApHyliiK5xCTNVSPiaAs=
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno=
github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A=
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
@@ -87,14 +109,20 @@ github.com/opentracing/opentracing-go v1.1.1-0.20190913142402-a7454ce5950e/go.mo
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/shirou/gopsutil v2.20.9+incompatible h1:msXs2frUV+O/JLva9EDLpuJ84PrFsdCTCQex8PUdtkQ=
github.com/shirou/gopsutil v2.20.9+incompatible/go.mod h1:5b4v6he4MtMOwMlS0TUMTu2PcXUg8+E1lC7eC3UO/RA=
github.com/shirou/gopsutil v3.21.5+incompatible h1:OloQyEerMi7JUrXiNzy8wQ5XN+baemxSl12QgIzt0jc=
github.com/shirou/gopsutil v3.21.5+incompatible/go.mod h1:5b4v6he4MtMOwMlS0TUMTu2PcXUg8+E1lC7eC3UO/RA=
github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72 h1:qLC7fQah7D6K1B0ujays3HV9gkFtllcxhzImRR7ArPQ=
github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
github.com/stretchr/testify v1.6.1 h1:hDPOHmpOpP40lSULcqw7IrRb/u7w6RpDC9399XyoNd0=
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/tklauser/go-sysconf v0.3.6 h1:oc1sJWvKkmvIxhDHeKWvZS4f6AW+YcoguSfRF2/Hmo4=
github.com/tklauser/go-sysconf v0.3.6/go.mod h1:MkWzOF4RMCshBAMXuhXJs64Rte09mITnppBXY/rYEFI=
github.com/tklauser/numcpus v0.2.2 h1:oyhllyrScuYI6g+h/zUvNXNp1wy7x8qQy3t/piefldA=
github.com/tklauser/numcpus v0.2.2/go.mod h1:x3qojaO3uyYt0i56EW/VUYs7uBvdl2fkfZFu0T9wgjM=
github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
go.opentelemetry.io/otel v0.7.0/go.mod h1:aZMyHG5TqDOXEgH2tyLiXSUKly1jT3yqE9PmrzIeCdo=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
@@ -102,13 +130,16 @@ golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20200513190911-00229845015e/go.mod h1:4M0jN8W1tt0AVLNr8HDosyJCDCDuyL9N9+3m7wDWgKw=
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b h1:+qEpEAPhDZ1o0x3tHzZTQDArnOixOzGD9HUJfcg0mb4=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20210508222113-6edffad5e616/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o=
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -117,12 +148,16 @@ golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7 h1:AeiKBIuRw3UomYXSbLy0Mc2dDLfdtbT/IVn4keq83P0=
golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM=
golang.org/x/net v0.0.0-20210614182718-04defd469f4e h1:XpT3nA5TvE525Ne3hInMh6+GETgn27Zfm9dxsThnX2Q=
golang.org/x/net v0.0.0-20210614182718-04defd469f4e/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -134,34 +169,52 @@ golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20191010194322-b09406accb47/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200519105757-fe76b779f299 h1:DYfZAGf2WMFjMxbgTjaC+2HC7NkNAQs+6Q8b9WEB/F4=
golang.org/x/sys v0.0.0-20200519105757-fe76b779f299/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210303074136-134d130e1a04/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210316164454-77fc1eacc6aa/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210616094352-59db8d763f22 h1:RqytpXGR1iVNX7psjB3ff8y7sNFinVFvkx1c8SjBkio=
golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.6 h1:aRYxNxv6iGQlyVaZmk6ZgYEDa+Jg18DxebPSrd6bg1M=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200207183749-b753a1ba74fa/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.1.3/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20191009194640-548a555dbc03/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013 h1:+kGHl1aib/qcwaRi1CbqBZ1rk19r85MNUf8HaBghugY=
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
google.golang.org/genproto v0.0.0-20210617175327-b9e0b3197ced h1:c5geK1iMU3cDKtFrCVQIcjR3W+JOZMuhIyICMCTbtus=
google.golang.org/genproto v0.0.0-20210617175327-b9e0b3197ced/go.mod h1:SzzZ/N+nwJDaO1kznhnlzqS8ocJICar6hYhVyhi++24=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.30.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.32.0 h1:zWTV+LMdc3kaiJMSTOFz2UgSBgx8RNQoTGiZu3fR9S0=
google.golang.org/grpc v1.32.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.38.0 h1:/9BgsAsa5nWe26HqOlvlgJnqBuktYOLCgjCPqsa56W0=
google.golang.org/grpc v1.38.0/go.mod h1:NREThFqKR1f3iQ6oBuvc5LadQuXVGo9rkm5ZGrQdJfM=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
@@ -170,15 +223,19 @@ google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzi
google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.25.0 h1:Ejskq+SyPohKW+1uil0JJMtmHCgJPJ/qWTxr8qp+R4c=
google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c=
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
google.golang.org/protobuf v1.26.0 h1:bxAC2xTBsZGibn2RTntX0oH50xLsqy1OxA9tTL3p/lk=
google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f h1:BLraFXnmrev5lT+xlilqcH8XK9/i0At2xKjWk4p6zsU=
gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.7/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=

View File

@@ -2,8 +2,11 @@ package apps
import (
"fmt"
"github.com/iwind/TeaGo/Tea"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/iwind/TeaGo/logs"
"github.com/iwind/TeaGo/maps"
"github.com/iwind/TeaGo/types"
"github.com/iwind/gosock/pkg/gosock"
"os"
"os/exec"
"runtime"
@@ -11,7 +14,7 @@ import (
"time"
)
// App命令帮助
// AppCmd App命令帮助
type AppCmd struct {
product string
version string
@@ -20,10 +23,14 @@ type AppCmd struct {
appendStrings []string
directives []*Directive
sock *gosock.Sock
}
func NewAppCmd() *AppCmd {
return &AppCmd{}
return &AppCmd{
sock: gosock.NewTmpSock(teaconst.ProcessName),
}
}
type CommandHelpOption struct {
@@ -31,25 +38,25 @@ type CommandHelpOption struct {
Description string
}
// 产品
// Product 产品
func (this *AppCmd) Product(product string) *AppCmd {
this.product = product
return this
}
// 版本
// Version 版本
func (this *AppCmd) Version(version string) *AppCmd {
this.version = version
return this
}
// 使用方法
// Usage 使用方法
func (this *AppCmd) Usage(usage string) *AppCmd {
this.usage = usage
return this
}
// 选项
// Option 选项
func (this *AppCmd) Option(code string, description string) *AppCmd {
this.options = append(this.options, &CommandHelpOption{
Code: code,
@@ -58,13 +65,13 @@ func (this *AppCmd) Option(code string, description string) *AppCmd {
return this
}
// 附加内容
// Append 附加内容
func (this *AppCmd) Append(appendString string) *AppCmd {
this.appendStrings = append(this.appendStrings, appendString)
return this
}
// 打印
// Print 打印
func (this *AppCmd) Print() {
fmt.Println(this.product + " v" + this.version)
@@ -103,7 +110,7 @@ func (this *AppCmd) Print() {
}
}
// 添加指令
// On 添加指令
func (this *AppCmd) On(arg string, callback func()) {
this.directives = append(this.directives, &Directive{
Arg: arg,
@@ -111,7 +118,7 @@ func (this *AppCmd) On(arg string, callback func()) {
})
}
// 运行
// Run 运行
func (this *AppCmd) Run(main func()) {
// 获取参数
args := os.Args[1:]
@@ -161,7 +168,7 @@ func (this *AppCmd) Run(main func()) {
// 版本号
func (this *AppCmd) runVersion() {
fmt.Println(this.product+" v"+this.version, "(build: "+runtime.Version(), runtime.GOOS, runtime.GOARCH+")")
fmt.Println(this.product+" v"+this.version, "(build: "+runtime.Version(), runtime.GOOS, runtime.GOARCH, teaconst.Tag+")")
}
// 帮助
@@ -171,9 +178,9 @@ func (this *AppCmd) runHelp() {
// 启动
func (this *AppCmd) runStart() {
proc := this.checkPid()
if proc != nil {
fmt.Println(this.product+" already started, pid:", proc.Pid)
var pid = this.getPID()
if pid > 0 {
fmt.Println(this.product+" already started, pid:", pid)
return
}
@@ -189,18 +196,15 @@ func (this *AppCmd) runStart() {
// 停止
func (this *AppCmd) runStop() {
proc := this.checkPid()
if proc == nil {
var pid = this.getPID()
if pid == 0 {
fmt.Println(this.product + " not started yet")
return
}
// 停止进程
_ = proc.Kill()
_, _ = this.sock.Send(&gosock.Command{Code: "stop"})
// 在Windows上经常不能及时释放资源
_ = DeletePid(Tea.Root + "/bin/pid")
fmt.Println(this.product+" stopped ok, pid:", proc.Pid)
fmt.Println(this.product+" stopped ok, pid:", types.String(pid))
}
// 重启
@@ -212,15 +216,24 @@ func (this *AppCmd) runRestart() {
// 状态
func (this *AppCmd) runStatus() {
proc := this.checkPid()
if proc == nil {
var pid = this.getPID()
if pid == 0 {
fmt.Println(this.product + " not started yet")
} else {
fmt.Println(this.product + " is running, pid: " + fmt.Sprintf("%d", proc.Pid))
return
}
fmt.Println(this.product + " is running, pid: " + types.String(pid))
}
// 检查PID
func (this *AppCmd) checkPid() *os.Process {
return CheckPid(Tea.Root + "/bin/pid")
// 获取当前的PID
func (this *AppCmd) getPID() int {
if !this.sock.IsListening() {
return 0
}
reply, err := this.sock.Send(&gosock.Command{Code: "pid"})
if err != nil {
return 0
}
return maps.NewMap(reply.Params).GetInt("pid")
}

View File

@@ -1,17 +0,0 @@
// +build !windows
package apps
import (
"os"
"syscall"
)
// lock file
func LockFile(fp *os.File) error {
return syscall.Flock(int(fp.Fd()), syscall.LOCK_EX|syscall.LOCK_NB)
}
func UnlockFile(fp *os.File) error {
return syscall.Flock(int(fp.Fd()), syscall.LOCK_UN)
}

View File

@@ -1,17 +0,0 @@
// +build windows
package apps
import (
"errors"
"os"
)
// lock file
func LockFile(fp *os.File) error {
return errors.New("not implemented on windows")
}
func UnlockFile(fp *os.File) error {
return errors.New("not implemented on windows")
}

View File

@@ -1,119 +0,0 @@
package apps
import (
"fmt"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/iwind/TeaGo/Tea"
"github.com/iwind/TeaGo/types"
"io/ioutil"
"os"
"runtime"
)
var pidFileList = []*os.File{}
// 检查Pid
func CheckPid(path string) *os.Process {
// windows上打开的文件是不能删除的
if runtime.GOOS == "windows" {
if os.Remove(path) == nil {
return nil
}
}
file, err := os.Open(path)
if err != nil {
return nil
}
defer func() {
_ = file.Close()
}()
// 是否能取得Lock
err = LockFile(file)
if err == nil {
_ = UnlockFile(file)
return nil
}
pidBytes, err := ioutil.ReadAll(file)
if err != nil {
return nil
}
pid := types.Int(string(pidBytes))
if pid <= 0 {
return nil
}
proc, _ := os.FindProcess(pid)
return proc
}
// 写入Pid
func WritePid() error {
path := Tea.Root + "/bin/pid"
fp, err := os.OpenFile(path, os.O_CREATE|os.O_TRUNC|os.O_WRONLY|os.O_RDONLY, 0666)
if err != nil {
return err
}
events.On(events.EventQuit, func() {
_ = fp.Close()
})
if runtime.GOOS != "windows" {
err = LockFile(fp)
if err != nil {
return err
}
}
pidFileList = append(pidFileList, fp) // hold the file pointers
_, err = fp.WriteString(fmt.Sprintf("%d", os.Getpid()))
if err != nil {
return err
}
return nil
}
// 写入Ppid
func WritePpid(path string) error {
fp, err := os.OpenFile(path, os.O_CREATE|os.O_TRUNC|os.O_WRONLY|os.O_RDONLY, 0666)
if err != nil {
return err
}
if runtime.GOOS != "windows" {
err = LockFile(fp)
if err != nil {
return err
}
}
pidFileList = append(pidFileList, fp) // hold the file pointers
_, err = fp.WriteString(fmt.Sprintf("%d", os.Getppid()))
if err != nil {
return err
}
return nil
}
// 删除Pid
func DeletePid(path string) error {
_, err := os.Stat(path)
if err != nil {
if !os.IsNotExist(err) {
return nil
}
return err
}
for _, fp := range pidFileList {
_ = UnlockFile(fp)
_ = fp.Close()
}
return os.Remove(path)
}

View File

@@ -5,6 +5,7 @@ package caches
import (
"database/sql"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/ttlcache"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/iwind/TeaGo/lists"
_ "github.com/mattn/go-sqlite3"
@@ -35,10 +36,15 @@ type FileList struct {
itemsTableName string
isClosed bool
memoryCache *ttlcache.Cache
}
func NewFileList(dir string) ListInterface {
return &FileList{dir: dir}
return &FileList{
dir: dir,
memoryCache: ttlcache.NewCache(),
}
}
func (this *FileList) Init() error {
@@ -54,7 +60,12 @@ func (this *FileList) Init() error {
this.itemsTableName = "cacheItems_v2"
db, err := sql.Open("sqlite3", "file:"+this.dir+"/index.db?cache=shared&mode=rwc&_journal_mode=WAL")
var dir = this.dir
if dir == "/" {
// 防止sqlite提示authority错误
dir = ""
}
db, err := sql.Open("sqlite3", "file:"+dir+"/index.db?cache=shared&mode=rwc&_journal_mode=WAL")
if err != nil {
return err
}
@@ -77,39 +88,7 @@ func (this *FileList) Init() error {
}**/
// 创建
_, err = db.Exec(`CREATE TABLE IF NOT EXISTS "` + this.itemsTableName + `" (
"id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
"hash" varchar(32),
"key" varchar(1024),
"headerSize" integer DEFAULT 0,
"bodySize" integer DEFAULT 0,
"metaSize" integer DEFAULT 0,
"expiredAt" integer DEFAULT 0,
"createdAt" integer DEFAULT 0,
"host" varchar(128),
"serverId" integer
);
CREATE INDEX IF NOT EXISTS "createdAt"
ON "` + this.itemsTableName + `" (
"createdAt" ASC
);
CREATE INDEX IF NOT EXISTS "expiredAt"
ON "` + this.itemsTableName + `" (
"expiredAt" ASC
);
CREATE UNIQUE INDEX IF NOT EXISTS "hash"
ON "` + this.itemsTableName + `" (
"hash" ASC
);
CREATE INDEX IF NOT EXISTS "serverId"
ON "` + this.itemsTableName + `" (
"serverId" ASC
);
`)
err = this.initTables(db, 1)
if err != nil {
return err
}
@@ -127,7 +106,7 @@ ON "` + this.itemsTableName + `" (
this.total = total
// 常用语句
this.existsByHashStmt, err = this.db.Prepare(`SELECT "bodySize" FROM "` + this.itemsTableName + `" WHERE "hash"=? AND expiredAt>? LIMIT 1`)
this.existsByHashStmt, err = this.db.Prepare(`SELECT "expiredAt" FROM "` + this.itemsTableName + `" WHERE "hash"=? AND expiredAt>? LIMIT 1`)
if err != nil {
return err
}
@@ -147,7 +126,7 @@ ON "` + this.itemsTableName + `" (
return err
}
this.statStmt, err = this.db.Prepare(`SELECT COUNT(*), IFNULL(SUM(headerSize+bodySize+metaSize), 0), IFNULL(SUM(headerSize+bodySize), 0) FROM "` + this.itemsTableName + `" WHERE expiredAt>?`)
this.statStmt, err = this.db.Prepare(`SELECT COUNT(*), IFNULL(SUM(headerSize+bodySize+metaSize), 0), IFNULL(SUM(headerSize+bodySize), 0) FROM "` + this.itemsTableName + `"`)
if err != nil {
return err
}
@@ -166,7 +145,7 @@ ON "` + this.itemsTableName + `" (
}
func (this *FileList) Reset() error {
// 不任何事情
// 不任何事情
return nil
}
@@ -193,6 +172,11 @@ func (this *FileList) Exist(hash string) (bool, error) {
return false, nil
}
item := this.memoryCache.Read(hash)
if item != nil {
return true, nil
}
rows, err := this.existsByHashStmt.Query(hash, time.Now().Unix())
if err != nil {
return false, err
@@ -201,6 +185,12 @@ func (this *FileList) Exist(hash string) (bool, error) {
_ = rows.Close()
}()
if rows.Next() {
var expiredAt int64
err = rows.Scan(&expiredAt)
if err != nil {
return false, nil
}
this.memoryCache.Write(hash, 1, expiredAt)
return true, nil
}
return false, nil
@@ -216,9 +206,13 @@ func (this *FileList) CleanPrefix(prefix string) error {
return nil
}
defer func() {
this.memoryCache.Clean()
}()
var count = int64(10000)
for {
result, err := this.db.Exec(`UPDATE "`+this.itemsTableName+`" SET expiredAt=0 WHERE id IN (SELECT id FROM "`+this.itemsTableName+`" WHERE expiredAt>0 AND createdAt<=? AND INSTR("key", ?)==1 LIMIT `+strconv.FormatInt(count, 10)+`)`, utils.UnixTime(), prefix)
result, err := this.db.Exec(`UPDATE "`+this.itemsTableName+`" SET expiredAt=0 WHERE id IN (SELECT id FROM "`+this.itemsTableName+`" WHERE expiredAt>0 AND createdAt<=? AND INSTR("key", ?)=1 LIMIT `+strconv.FormatInt(count, 10)+`)`, utils.UnixTime(), prefix)
if err != nil {
return err
}
@@ -237,6 +231,9 @@ func (this *FileList) Remove(hash string) error {
return nil
}
// 从缓存中删除
this.memoryCache.Delete(hash)
row := this.selectByHashStmt.QueryRow(hash)
if row.Err() != nil {
return row.Err()
@@ -281,19 +278,18 @@ func (this *FileList) Purge(count int, callback func(hash string) error) error {
if err != nil {
return err
}
defer func() {
_ = rows.Close()
}()
hashStrings := []string{}
for rows.Next() {
var hash string
err = rows.Scan(&hash)
if err != nil {
_ = rows.Close()
return err
}
hashStrings = append(hashStrings, hash)
}
_ = rows.Close() // 不能使用defer防止读写冲突
// 不在 rows.Next() 循环中操作是为了避免死锁
for _, hash := range hashStrings {
@@ -316,6 +312,8 @@ func (this *FileList) CleanAll() error {
return nil
}
this.memoryCache.Clean()
_, err := this.deleteAllStmt.Exec()
if err != nil {
return err
@@ -330,7 +328,7 @@ func (this *FileList) Stat(check func(hash string) bool) (*Stat, error) {
}
// 这里不设置过期时间、不使用 check 函数,目的是让查询更快速一些
row := this.statStmt.QueryRow(time.Now().Unix())
row := this.statStmt.QueryRow()
if row.Err() != nil {
return nil, row.Err()
}
@@ -362,6 +360,8 @@ func (this *FileList) OnRemove(f func(item *Item)) {
func (this *FileList) Close() error {
this.isClosed = true
this.memoryCache.Destroy()
if this.db != nil {
_ = this.existsByHashStmt.Close()
_ = this.insertStmt.Close()
@@ -376,6 +376,58 @@ func (this *FileList) Close() error {
return nil
}
// 初始化
func (this *FileList) initTables(db *sql.DB, times int) error {
_, err := db.Exec(`CREATE TABLE IF NOT EXISTS "` + this.itemsTableName + `" (
"id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
"hash" varchar(32),
"key" varchar(1024),
"headerSize" integer DEFAULT 0,
"bodySize" integer DEFAULT 0,
"metaSize" integer DEFAULT 0,
"expiredAt" integer DEFAULT 0,
"createdAt" integer DEFAULT 0,
"host" varchar(128),
"serverId" integer
);
CREATE INDEX IF NOT EXISTS "createdAt"
ON "` + this.itemsTableName + `" (
"createdAt" ASC
);
CREATE INDEX IF NOT EXISTS "expiredAt"
ON "` + this.itemsTableName + `" (
"expiredAt" ASC
);
CREATE UNIQUE INDEX IF NOT EXISTS "hash"
ON "` + this.itemsTableName + `" (
"hash" ASC
);
CREATE INDEX IF NOT EXISTS "serverId"
ON "` + this.itemsTableName + `" (
"serverId" ASC
);
`)
if err != nil {
// 尝试删除重建
if times < 3 {
_, dropErr := db.Exec(`DROP TABLE "` + this.itemsTableName + `"`)
if dropErr == nil {
return this.initTables(db, times+1)
}
return err
}
return err
}
return nil
}
// 删除过期不用的表格
func (this *FileList) removeOldTables() error {
rows, err := this.db.Query(`SELECT "name" FROM sqlite_master WHERE "type"='table'`)
if err != nil {

View File

@@ -50,7 +50,7 @@ func TestFileList_Add_Many(t *testing.T) {
}
before := time.Now()
for i := 0; i < 2000_0000; i++ {
u := "http://edge.teaos.cn/123456" + strconv.Itoa(i)
u := "https://edge.teaos.cn/123456" + strconv.Itoa(i)
_ = list.Add(stringutil.Md5(u), &Item{
Key: u,
ExpiredAt: time.Now().Unix() + 3600,
@@ -236,6 +236,27 @@ func TestFileList_CleanAll(t *testing.T) {
t.Log(list.Count())
}
func TestFileList_Conflict(t *testing.T) {
list := NewFileList(Tea.Root + "/data").(*FileList)
err := list.Init()
if err != nil {
t.Fatal(err)
}
rows, err := list.purgeStmt.Query(time.Now().Unix(), 1000)
if err != nil {
t.Fatal(err)
}
go func() {
time.Sleep(5 * time.Second)
_ = rows.Close()
}()
t.Log("before exists")
t.Log(list.Exist("123456"))
t.Log("after exists")
}
func BenchmarkFileList_Exist(b *testing.B) {
list := NewFileList(Tea.Root + "/data")
err := list.Init()

View File

@@ -5,6 +5,7 @@ import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs/shared"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/iwind/TeaGo/lists"
"github.com/iwind/TeaGo/types"
"strconv"
"sync"
)
@@ -162,3 +163,25 @@ func (this *Manager) TotalMemorySize() int64 {
}
return total
}
// FindAllCachePaths 所有缓存路径
func (this *Manager) FindAllCachePaths() []string {
this.locker.Lock()
defer this.locker.Unlock()
var result = []string{}
for _, policy := range this.policyMap {
if policy.Type == serverconfigs.CachePolicyStorageFile {
if policy.Options != nil {
dir, ok := policy.Options["dir"]
if ok {
var dirString = types.String(dir)
if len(dirString) > 0 {
result = append(result, dirString)
}
}
}
}
}
return result
}

View File

@@ -34,13 +34,7 @@ func (this *FileReader) Init() error {
}
}()
// 读取状态
_, err := this.fp.Seek(SizeExpiresAt, io.SeekStart)
if err != nil {
_ = this.discard()
return err
}
buf := make([]byte, 3)
var buf = make([]byte, SizeMeta)
ok, err := this.readToBuff(this.fp, buf)
if err != nil {
return err
@@ -48,37 +42,18 @@ func (this *FileReader) Init() error {
if !ok {
return ErrNotFound
}
status := types.Int(string(buf))
status := types.Int(string(buf[SizeExpiresAt : SizeExpiresAt+SizeStatus]))
if status < 100 || status > 999 {
return errors.New("invalid status")
}
this.status = status
// URL
_, err = this.fp.Seek(SizeExpiresAt+SizeStatus, io.SeekStart)
if err != nil {
return err
}
bytes4 := make([]byte, 4)
ok, err = this.readToBuff(this.fp, bytes4)
if err != nil {
return err
}
if !ok {
return ErrNotFound
}
urlLength := binary.BigEndian.Uint32(bytes4)
urlLength := binary.BigEndian.Uint32(buf[SizeExpiresAt+SizeStatus : SizeExpiresAt+SizeStatus+SizeURLLength])
// header
ok, err = this.readToBuff(this.fp, bytes4)
if err != nil {
return err
}
if !ok {
return ErrNotFound
}
headerSize := int(binary.BigEndian.Uint32(bytes4))
headerSize := int(binary.BigEndian.Uint32(buf[SizeExpiresAt+SizeStatus+SizeURLLength : SizeExpiresAt+SizeStatus+SizeURLLength+SizeHeaderLength]))
if headerSize == 0 {
return nil
}
@@ -86,16 +61,9 @@ func (this *FileReader) Init() error {
this.headerOffset = int64(SizeMeta) + int64(urlLength)
// body
bytes8 := make([]byte, 8)
ok, err = this.readToBuff(this.fp, bytes8)
if err != nil {
return err
}
if !ok {
return ErrNotFound
}
bodySize := int(binary.BigEndian.Uint64(bytes8))
bodySize := int(binary.BigEndian.Uint64(buf[SizeExpiresAt+SizeStatus+SizeURLLength+SizeHeaderLength : SizeExpiresAt+SizeStatus+SizeURLLength+SizeHeaderLength+SizeBodyLength]))
if bodySize == 0 {
isOk = true
return nil
}
this.bodySize = int64(bodySize)

View File

@@ -49,6 +49,9 @@ func TestFileReader(t *testing.T) {
t.Log("body:", string(buf[:n]))
return true, nil
})
if err != nil {
t.Fatal(err)
}
}
func TestFileReader_Range(t *testing.T) {

View File

@@ -499,29 +499,12 @@ func (this *FileStorage) CleanAll() error {
}
// 重新遍历待删除
fp2, err := os.Open(dir)
if err != nil {
return err
}
defer func() {
_ = fp2.Close()
}()
subDirs, err = fp2.Readdir(-1)
if err != nil {
return err
}
for _, info := range subDirs {
subDir := info.Name()
if !strings.HasSuffix(subDir, "-deleted") {
continue
}
// 删除
err = os.RemoveAll(dir + "/" + subDir)
go func() {
err = this.cleanDeletedDirs(dir)
if err != nil {
return err
remotelogs.Warn("CACHE", "delete '*-deleted' dirs failed: "+err.Error())
}
}
}()
return nil
}
@@ -756,7 +739,7 @@ func (this *FileStorage) purgeLoop() {
return nil
})
if err != nil {
remotelogs.Warn("CACHE", "purge file storage failed: " + err.Error())
remotelogs.Warn("CACHE", "purge file storage failed: "+err.Error())
}
}
@@ -798,3 +781,34 @@ func (this *FileStorage) diskCapacityBytes() int64 {
}
return c1
}
// 清理 *-deleted 目录
// 由于在很多硬盘上耗时非常久,所以应该放在后台运行
func (this *FileStorage) cleanDeletedDirs(dir string) error {
fp, err := os.Open(dir)
if err != nil {
return err
}
defer func() {
_ = fp.Close()
}()
subDirs, err := fp.Readdir(-1)
if err != nil {
return err
}
for _, info := range subDirs {
subDir := info.Name()
if !strings.HasSuffix(subDir, "-deleted") {
continue
}
// 删除
err = os.RemoveAll(dir + "/" + subDir)
if err != nil {
if !os.IsNotExist(err) {
return err
}
}
}
return nil
}

View File

@@ -77,10 +77,9 @@ func (this *MemoryStorage) OpenReader(key string) (Reader, error) {
hash := this.hash(key)
this.locker.RLock()
defer this.locker.RUnlock()
item := this.valuesMap[hash]
if item == nil || !item.IsDone {
this.locker.RUnlock()
return nil, ErrNotFound
}
@@ -88,10 +87,13 @@ func (this *MemoryStorage) OpenReader(key string) (Reader, error) {
reader := NewMemoryReader(item)
err := reader.Init()
if err != nil {
this.locker.RUnlock()
return nil, err
}
this.locker.RUnlock()
return reader, nil
}
this.locker.RUnlock()
_ = this.Delete(key)
@@ -137,7 +139,7 @@ func (this *MemoryStorage) OpenWriter(key string, expiredAt int64, status int) (
}
// 先删除
err = this.deleteWithoutKey(key)
err = this.deleteWithoutLocker(key)
if err != nil {
return nil, err
}
@@ -276,7 +278,7 @@ func (this *MemoryStorage) memoryCapacityBytes() int64 {
return c1
}
func (this *MemoryStorage) deleteWithoutKey(key string) error {
func (this *MemoryStorage) deleteWithoutLocker(key string) error {
hash := this.hash(key)
delete(this.valuesMap, hash)
_ = this.list.Remove(fmt.Sprintf("%d", hash))

View File

@@ -84,6 +84,19 @@ func TestMemoryStorage_OpenWriter(t *testing.T) {
}
}
func TestMemoryStorage_OpenReaderLock(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{})
_ = storage.Init()
var h = storage.hash("test")
storage.valuesMap = map[uint64]*MemoryItem{
h: {
IsDone: true,
},
}
_, _ = storage.OpenReader("test")
}
func TestMemoryStorage_Delete(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{})
{
@@ -237,3 +250,18 @@ func TestMemoryStorage_Expire(t *testing.T) {
}
time.Sleep(70 * time.Second)
}
func TestMemoryStorage_Locker(t *testing.T) {
storage := NewMemoryStorage(&serverconfigs.HTTPCachePolicy{})
err := storage.Init()
if err != nil {
t.Fatal(err)
}
storage.locker.Lock()
err = storage.deleteWithoutLocker("a")
storage.locker.Unlock()
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}

View File

@@ -1,31 +1,31 @@
package caches
// 缓存内容写入接口
// Writer 缓存内容写入接口
type Writer interface {
// 写入Header数据
// WriteHeader 写入Header数据
WriteHeader(data []byte) (n int, err error)
// 写入Body数据
// Write 写入Body数据
Write(data []byte) (n int, err error)
// 写入的Header数据大小
// HeaderSize 写入的Header数据大小
HeaderSize() int64
// 写入的Body数据大小
// BodySize 写入的Body数据大小
BodySize() int64
// 关闭
// Close 关闭
Close() error
// 丢弃
// Discard 丢弃
Discard() error
// Key
// Key Key
Key() string
// 过期时间
// ExpiredAt 过期时间
ExpiredAt() int64
// 内容类型
// ItemType 内容类型
ItemType() ItemType
}

View File

@@ -0,0 +1,76 @@
package caches
import (
"github.com/TeaOSLab/EdgeNode/internal/compressions"
)
type compressionWriter struct {
rawWriter Writer
writer compressions.Writer
key string
expiredAt int64
}
func NewCompressionWriter(gw Writer, cpWriter compressions.Writer, key string, expiredAt int64) Writer {
return &compressionWriter{
rawWriter: gw,
writer: cpWriter,
key: key,
expiredAt: expiredAt,
}
}
func (this *compressionWriter) WriteHeader(data []byte) (n int, err error) {
return this.writer.Write(data)
}
// WriteHeaderLength 写入Header长度数据
func (this *compressionWriter) WriteHeaderLength(headerLength int) error {
return nil
}
// WriteBodyLength 写入Body长度数据
func (this *compressionWriter) WriteBodyLength(bodyLength int64) error {
return nil
}
func (this *compressionWriter) Write(data []byte) (n int, err error) {
return this.writer.Write(data)
}
func (this *compressionWriter) Close() error {
err := this.writer.Close()
if err != nil {
return err
}
return this.rawWriter.Close()
}
func (this *compressionWriter) Discard() error {
err := this.writer.Close()
if err != nil {
return err
}
return this.rawWriter.Discard()
}
func (this *compressionWriter) Key() string {
return this.key
}
func (this *compressionWriter) ExpiredAt() int64 {
return this.expiredAt
}
func (this *compressionWriter) HeaderSize() int64 {
return this.rawWriter.HeaderSize()
}
func (this *compressionWriter) BodySize() int64 {
return this.rawWriter.BodySize()
}
// ItemType 内容类型
func (this *compressionWriter) ItemType() ItemType {
return this.rawWriter.ItemType()
}

View File

@@ -1,76 +0,0 @@
package caches
import (
"compress/gzip"
)
type gzipWriter struct {
rawWriter Writer
writer *gzip.Writer
key string
expiredAt int64
}
func NewGzipWriter(gw Writer, key string, expiredAt int64) Writer {
return &gzipWriter{
rawWriter: gw,
writer: gzip.NewWriter(gw),
key: key,
expiredAt: expiredAt,
}
}
func (this *gzipWriter) WriteHeader(data []byte) (n int, err error) {
return this.writer.Write(data)
}
// 写入Header长度数据
func (this *gzipWriter) WriteHeaderLength(headerLength int) error {
return nil
}
// 写入Body长度数据
func (this *gzipWriter) WriteBodyLength(bodyLength int64) error {
return nil
}
func (this *gzipWriter) Write(data []byte) (n int, err error) {
return this.writer.Write(data)
}
func (this *gzipWriter) Close() error {
err := this.writer.Close()
if err != nil {
return err
}
return this.rawWriter.Close()
}
func (this *gzipWriter) Discard() error {
err := this.writer.Close()
if err != nil {
return err
}
return this.rawWriter.Discard()
}
func (this *gzipWriter) Key() string {
return this.key
}
func (this *gzipWriter) ExpiredAt() int64 {
return this.expiredAt
}
func (this *gzipWriter) HeaderSize() int64 {
return this.rawWriter.HeaderSize()
}
func (this *gzipWriter) BodySize() int64 {
return this.rawWriter.BodySize()
}
// 内容类型
func (this *gzipWriter) ItemType() ItemType {
return this.rawWriter.ItemType()
}

View File

@@ -0,0 +1,21 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
import (
"errors"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"io"
)
func NewWriter(writer io.Writer, compressType serverconfigs.HTTPCompressionType, level int) (Writer, error) {
switch compressType {
case serverconfigs.HTTPCompressionTypeGzip:
return NewGzipWriter(writer, level)
case serverconfigs.HTTPCompressionTypeDeflate:
return NewDeflateWriter(writer, level)
case serverconfigs.HTTPCompressionTypeBrotli:
return NewBrotliWriter(writer, level)
}
return nil, errors.New("invalid compression type '" + compressType + "'")
}

View File

@@ -0,0 +1,10 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
type Writer interface {
Write(p []byte) (int, error)
Flush() error
Close() error
Level() int
}

View File

@@ -0,0 +1,41 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
import (
"github.com/andybalholm/brotli"
"io"
)
type BrotliWriter struct {
writer *brotli.Writer
level int
}
func NewBrotliWriter(writer io.Writer, level int) (Writer, error) {
if level <= 0 {
level = brotli.BestSpeed
} else if level > brotli.BestCompression {
level = brotli.BestCompression
}
return &BrotliWriter{
writer: brotli.NewWriterLevel(writer, level),
level: level,
}, nil
}
func (this *BrotliWriter) Write(p []byte) (int, error) {
return this.writer.Write(p)
}
func (this *BrotliWriter) Flush() error {
return this.writer.Flush()
}
func (this *BrotliWriter) Close() error {
return this.writer.Close()
}
func (this *BrotliWriter) Level() int {
return this.level
}

View File

@@ -0,0 +1,47 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
import (
"compress/flate"
"io"
)
type DeflateWriter struct {
writer *flate.Writer
level int
}
func NewDeflateWriter(writer io.Writer, level int) (Writer, error) {
if level <= 0 {
level = flate.BestSpeed
} else if level > flate.BestCompression {
level = flate.BestCompression
}
flateWriter, err := flate.NewWriter(writer, level)
if err != nil {
return nil, err
}
return &DeflateWriter{
writer: flateWriter,
level: level,
}, nil
}
func (this *DeflateWriter) Write(p []byte) (int, error) {
return this.writer.Write(p)
}
func (this *DeflateWriter) Flush() error {
return this.writer.Flush()
}
func (this *DeflateWriter) Close() error {
return this.writer.Close()
}
func (this *DeflateWriter) Level() int {
return this.level
}

View File

@@ -0,0 +1,47 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package compressions
import (
"compress/gzip"
"io"
)
type GzipWriter struct {
writer *gzip.Writer
level int
}
func NewGzipWriter(writer io.Writer, level int) (Writer, error) {
if level <= 0 {
level = gzip.BestSpeed
} else if level > gzip.BestCompression {
level = gzip.BestCompression
}
gzipWriter, err := gzip.NewWriterLevel(writer, level)
if err != nil {
return nil, err
}
return &GzipWriter{
writer: gzipWriter,
level: level,
}, nil
}
func (this *GzipWriter) Write(p []byte) (int, error) {
return this.writer.Write(p)
}
func (this *GzipWriter) Flush() error {
return this.writer.Flush()
}
func (this *GzipWriter) Close() error {
return this.writer.Close()
}
func (this *GzipWriter) Level() int {
return this.level
}

8
internal/const/build.go Normal file
View File

@@ -0,0 +1,8 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
// +build community
package teaconst
const BuildCommunity = true
const BuildPlus = false
const Tag = "community"

View File

@@ -0,0 +1,8 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
// +build plus
package teaconst
const BuildCommunity = false
const BuildPlus = true
const Tag = "plus"

View File

@@ -1,7 +1,7 @@
package teaconst
const (
Version = "0.2.0"
Version = "0.3.2"
ProductName = "Edge Node"
ProcessName = "edge-node"

View File

@@ -10,7 +10,7 @@ const (
IPItemTypeAll IPItemType = "all" // 所有IP
)
// IP条目
// IPItem IP条目
type IPItem struct {
Type string `json:"type"`
Id int64 `json:"id"`
@@ -20,7 +20,7 @@ type IPItem struct {
EventLevel string `json:"eventLevel"`
}
// 检查是否包含某个IP
// Contains 检查是否包含某个IP
func (this *IPItem) Contains(ip uint64) bool {
switch this.Type {
case IPItemTypeIPv4:

View File

@@ -3,6 +3,7 @@ package iplibrary
import (
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/iwind/TeaGo/assert"
"runtime"
"testing"
"time"
)
@@ -72,3 +73,36 @@ func TestIPItem_Contains(t *testing.T) {
a.IsTrue(item.Contains(utils.IP2Long("192.168.1.1")))
}
}
func TestIPItem_Memory(t *testing.T) {
var list = NewIPList()
for i := 0; i < 2_000_000; i ++ {
list.Add(&IPItem{
Type: "ip",
Id: int64(i),
IPFrom: utils.IP2Long("192.168.1.1"),
IPTo: 0,
ExpiredAt: time.Now().Unix(),
EventLevel: "",
})
}
t.Log("waiting")
time.Sleep(10 * time.Second)
}
func BenchmarkIPItem_Contains(b *testing.B) {
runtime.GOMAXPROCS(1)
item := &IPItem{
IPFrom: utils.IP2Long("192.168.1.1"),
IPTo: utils.IP2Long("192.168.1.101"),
ExpiredAt: 0,
}
ip := utils.IP2Long("192.168.1.1")
for i := 0; i < b.N; i++ {
for j := 0; j < 10_000; j++ {
item.Contains(ip)
}
}
}

View File

@@ -3,24 +3,26 @@ package iplibrary
import (
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/TeaOSLab/EdgeNode/internal/utils/expires"
"sort"
"sync"
)
// IP名单
// IPList IP名单
// TODO IP名单可以分片关闭这样让每一片的数据量减少查询更快
type IPList struct {
itemsMap map[int64]*IPItem // id => item
ipMap map[uint64][]int64 // ip => itemIds
expireList *expires.List
itemsMap map[int64]*IPItem // id => item
sortedItems []*IPItem
allItemsMap map[int64]*IPItem // id => item
isAll bool
expireList *expires.List
locker sync.RWMutex
}
func NewIPList() *IPList {
list := &IPList{
itemsMap: map[int64]*IPItem{},
ipMap: map[uint64][]int64{},
itemsMap: map[int64]*IPItem{},
allItemsMap: map[int64]*IPItem{},
}
expireList := expires.NewList()
@@ -34,14 +36,94 @@ func NewIPList() *IPList {
}
func (this *IPList) Add(item *IPItem) {
this.addItem(item, true)
}
// AddDelay 延迟添加需要手工调用Sort()函数
func (this *IPList) AddDelay(item *IPItem) {
this.addItem(item, false)
}
func (this *IPList) Sort() {
this.locker.Lock()
this.sortItems()
this.locker.Unlock()
}
func (this *IPList) Delete(itemId int64) {
this.locker.Lock()
this.deleteItem(itemId)
this.locker.Unlock()
}
// Contains 判断是否包含某个IP
func (this *IPList) Contains(ip uint64) bool {
this.locker.RLock()
if len(this.allItemsMap) > 0 {
this.locker.RUnlock()
return true
}
var item = this.lookupIP(ip)
this.locker.RUnlock()
return item != nil
}
// ContainsIPStrings 是否包含一组IP中的任意一个并返回匹配的第一个Item
func (this *IPList) ContainsIPStrings(ipStrings []string) (item *IPItem, found bool) {
if len(ipStrings) == 0 {
return
}
this.locker.RLock()
if len(this.allItemsMap) > 0 {
for _, allItem := range this.allItemsMap {
item = allItem
break
}
if item != nil {
this.locker.RUnlock()
found = true
return
}
}
for _, ipString := range ipStrings {
if len(ipString) == 0 {
continue
}
item = this.lookupIP(utils.IP2Long(ipString))
if item != nil {
this.locker.RUnlock()
found = true
return
}
}
this.locker.RUnlock()
return
}
func (this *IPList) addItem(item *IPItem, sortable bool) {
if item == nil {
return
}
if item.ExpiredAt > 0 && item.ExpiredAt < utils.UnixTime() {
return
}
if item.IPFrom == 0 && item.IPTo == 0 {
if item.Type != "all" {
if item.Type != IPItemTypeAll {
return
}
} else if item.IPTo > 0 {
if item.IPFrom > item.IPTo {
item.IPFrom, item.IPTo = item.IPTo, item.IPFrom
} else if item.IPFrom == 0 {
item.IPFrom = item.IPTo
item.IPTo = 0
}
}
this.locker.Lock()
@@ -56,157 +138,86 @@ func (this *IPList) Add(item *IPItem) {
// 展开
if item.IPFrom > 0 {
if item.IPTo == 0 {
this.addIP(item.IPFrom, item.Id)
} else {
if item.IPFrom > item.IPTo {
item.IPTo, item.IPFrom = item.IPFrom, item.IPTo
}
for i := item.IPFrom; i <= item.IPTo; i++ {
// 最多不能超过65535防止整个系统内存爆掉
if i >= item.IPFrom+65535 {
break
}
this.addIP(i, item.Id)
}
}
} else if item.IPTo > 0 {
this.addIP(item.IPTo, item.Id)
this.sortedItems = append(this.sortedItems, item)
} else {
this.addIP(0, item.Id)
// 更新isAll
this.isAll = true
this.allItemsMap[item.Id] = item
}
if item.ExpiredAt > 0 {
this.expireList.Add(item.Id, item.ExpiredAt)
}
if sortable {
this.sortItems()
}
this.locker.Unlock()
}
func (this *IPList) Delete(itemId int64) {
this.locker.Lock()
defer this.locker.Unlock()
this.deleteItem(itemId)
// 更新isAll
this.isAll = len(this.ipMap[0]) > 0
// 对列表进行排序
func (this *IPList) sortItems() {
sort.Slice(this.sortedItems, func(i, j int) bool {
var item1 = this.sortedItems[i]
var item2 = this.sortedItems[j]
if item1.IPFrom == item2.IPFrom {
return item1.IPTo < item2.IPTo
}
return item1.IPFrom < item2.IPFrom
})
}
// 判断是否包含某个IP
func (this *IPList) Contains(ip uint64) bool {
this.locker.RLock()
if this.isAll {
this.locker.RUnlock()
return true
}
_, ok := this.ipMap[ip]
this.locker.RUnlock()
return ok
}
// 是否包含一组IP
func (this *IPList) ContainsIPStrings(ipStrings []string) (found bool, item *IPItem) {
if len(ipStrings) == 0 {
return
}
this.locker.RLock()
if this.isAll {
itemIds := this.ipMap[0]
if len(itemIds) > 0 {
itemId := itemIds[0]
item = this.itemsMap[itemId]
}
this.locker.RUnlock()
found = true
return
}
for _, ipString := range ipStrings {
if len(ipString) == 0 {
continue
}
itemIds, ok := this.ipMap[utils.IP2Long(ipString)]
if ok {
if len(itemIds) > 0 {
itemId := itemIds[0]
item = this.itemsMap[itemId]
// 不加锁的情况下查找Item
func (this *IPList) lookupIP(ip uint64) *IPItem {
var count = len(this.sortedItems)
var resultIndex = -1
sort.Search(count, func(i int) bool {
var item = this.sortedItems[i]
if item.IPFrom < ip {
if item.IPTo >= ip {
resultIndex = i
}
this.locker.RUnlock()
found = true
return
return false
} else if item.IPFrom == ip {
resultIndex = i
return false
}
return true
})
if resultIndex < 0 || resultIndex >= count {
return nil
}
this.locker.RUnlock()
return
return this.sortedItems[resultIndex]
}
// 在不加锁的情况下删除某个Item
// 将会被别的方法引用,切记不能加锁
func (this *IPList) deleteItem(itemId int64) {
item, ok := this.itemsMap[itemId]
_, ok := this.itemsMap[itemId]
if !ok {
return
}
delete(this.itemsMap, itemId)
// 展开
if item.IPFrom > 0 {
if item.IPTo == 0 {
this.deleteIP(item.IPFrom, item.Id)
} else {
if item.IPFrom > item.IPTo {
item.IPTo, item.IPFrom = item.IPFrom, item.IPTo
}
for i := item.IPFrom; i <= item.IPTo; i++ {
// 最多不能超过65535防止整个系统内存爆掉
if i >= item.IPFrom+65535 {
break
}
this.deleteIP(i, item.Id)
}
}
} else if item.IPTo > 0 {
this.deleteIP(item.IPTo, item.Id)
} else {
this.deleteIP(0, item.Id)
}
}
// 添加单个IP
func (this *IPList) addIP(ip uint64, itemId int64) {
itemIds, ok := this.ipMap[ip]
// 是否为All Item
_, ok = this.allItemsMap[itemId]
if ok {
itemIds = append(itemIds, itemId)
} else {
itemIds = []int64{itemId}
}
this.ipMap[ip] = itemIds
}
// 删除单个IP
func (this *IPList) deleteIP(ip uint64, itemId int64) {
itemIds, ok := this.ipMap[ip]
if !ok {
delete(this.allItemsMap, itemId)
return
}
newItemIds := []int64{}
for _, oldItemId := range itemIds {
if oldItemId == itemId {
continue
// 删除排序中的Item
var index = -1
for itemIndex, item := range this.sortedItems {
if item.Id == itemId {
index = itemIndex
break
}
newItemIds = append(newItemIds, oldItemId)
}
if len(newItemIds) > 0 {
this.ipMap[ip] = newItemIds
} else {
delete(this.ipMap, ip)
if index >= 0 {
copy(this.sortedItems[index:], this.sortedItems[index+1:])
this.sortedItems = this.sortedItems[:len(this.sortedItems)-1]
}
}

View File

@@ -4,6 +4,7 @@ import (
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/iwind/TeaGo/assert"
"github.com/iwind/TeaGo/logs"
"github.com/iwind/TeaGo/rands"
"runtime"
"strconv"
"testing"
@@ -16,7 +17,7 @@ func TestIPList_Add_Empty(t *testing.T) {
Id: 1,
})
logs.PrintAsJSON(ipList.itemsMap, t)
logs.PrintAsJSON(ipList.ipMap, t)
logs.PrintAsJSON(ipList.allItemsMap, t)
}
func TestIPList_Add_One(t *testing.T) {
@@ -31,15 +32,30 @@ func TestIPList_Add_One(t *testing.T) {
})
ipList.Add(&IPItem{
Id: 3,
IPFrom: utils.IP2Long("2001:db8:0:1::101"),
IPFrom: utils.IP2Long("192.168.0.2"),
})
ipList.Add(&IPItem{
Id: 4,
IPFrom: utils.IP2Long("192.168.0.2"),
IPTo: utils.IP2Long("192.168.0.1"),
})
ipList.Add(&IPItem{
Id: 5,
IPFrom: utils.IP2Long("2001:db8:0:1::101"),
})
ipList.Add(&IPItem{
Id: 6,
IPFrom: 0,
Type: "all",
})
t.Log("===items===")
logs.PrintAsJSON(ipList.itemsMap, t)
logs.PrintAsJSON(ipList.ipMap, t) // ip => items
t.Log("===sorted items===")
logs.PrintAsJSON(ipList.sortedItems, t)
t.Log("===all items===")
logs.PrintAsJSON(ipList.allItemsMap, t) // ip => items
}
func TestIPList_Update(t *testing.T) {
@@ -50,14 +66,31 @@ func TestIPList_Update(t *testing.T) {
})
/**ipList.Add(&IPItem{
Id: 2,
IPFrom: IP2Long("192.168.1.1"),
IPFrom: utils.IP2Long("192.168.1.1"),
})**/
ipList.Add(&IPItem{
Id: 1,
IPTo: utils.IP2Long("192.168.1.2"),
})
logs.PrintAsJSON(ipList.itemsMap, t)
logs.PrintAsJSON(ipList.ipMap, t)
logs.PrintAsJSON(ipList.sortedItems, t)
}
func TestIPList_Update_AllItems(t *testing.T) {
ipList := NewIPList()
ipList.Add(&IPItem{
Id: 1,
Type: IPItemTypeAll,
IPFrom: 0,
})
ipList.Add(&IPItem{
Id: 1,
IPTo: 0,
})
t.Log("===items map===")
logs.PrintAsJSON(ipList.itemsMap, t)
t.Log("===all items map===")
logs.PrintAsJSON(ipList.allItemsMap, t)
}
func TestIPList_Add_Range(t *testing.T) {
@@ -71,9 +104,9 @@ func TestIPList_Add_Range(t *testing.T) {
Id: 2,
IPTo: utils.IP2Long("192.168.1.2"),
})
t.Log(len(ipList.ipMap), "ips")
t.Log(len(ipList.itemsMap), "ips")
logs.PrintAsJSON(ipList.itemsMap, t)
logs.PrintAsJSON(ipList.ipMap, t)
logs.PrintAsJSON(ipList.allItemsMap, t)
}
func TestIPList_Add_Overflow(t *testing.T) {
@@ -85,8 +118,8 @@ func TestIPList_Add_Overflow(t *testing.T) {
IPFrom: utils.IP2Long("192.168.1.1"),
IPTo: utils.IP2Long("192.169.255.1"),
})
t.Log(len(ipList.ipMap), "ips")
a.IsTrue(len(ipList.ipMap) <= 65535)
t.Log(len(ipList.itemsMap), "ips")
a.IsTrue(len(ipList.itemsMap) <= 65535)
}
func TestNewIPList_Memory(t *testing.T) {
@@ -104,20 +137,50 @@ func TestNewIPList_Memory(t *testing.T) {
}
func TestIPList_Contains(t *testing.T) {
var a = assert.NewAssertion(t)
list := NewIPList()
for i := 0; i < 255; i++ {
list.Add(&IPItem{
list.AddDelay(&IPItem{
Id: int64(i),
IPFrom: utils.IP2Long(strconv.Itoa(i) + ".168.0.1"),
IPTo: utils.IP2Long(strconv.Itoa(i) + ".168.255.1"),
ExpiredAt: 0,
})
}
t.Log(len(list.ipMap), "ip")
for i := 0; i < 255; i++ {
list.AddDelay(&IPItem{
Id: int64(1000 + i),
IPFrom: utils.IP2Long("192.167.2." + strconv.Itoa(i)),
})
}
list.Sort()
t.Log(len(list.itemsMap), "ip")
before := time.Now()
t.Log(list.Contains(utils.IP2Long("192.168.1.100")))
t.Log(list.Contains(utils.IP2Long("192.168.2.100")))
a.IsTrue(list.Contains(utils.IP2Long("192.168.1.100")))
a.IsTrue(list.Contains(utils.IP2Long("192.168.2.100")))
a.IsFalse(list.Contains(utils.IP2Long("192.169.3.100")))
a.IsFalse(list.Contains(utils.IP2Long("192.167.3.100")))
a.IsTrue(list.Contains(utils.IP2Long("192.167.2.100")))
t.Log(time.Since(before).Seconds()*1000, "ms")
}
func TestIPList_Contains_Many(t *testing.T) {
list := NewIPList()
for i := 0; i < 1_000_000; i++ {
list.AddDelay(&IPItem{
Id: int64(i),
IPFrom: utils.IP2Long(strconv.Itoa(rands.Int(0, 255)) + "." + strconv.Itoa(rands.Int(0, 255)) + "." + strconv.Itoa(rands.Int(0, 255)) + "." + strconv.Itoa(rands.Int(0, 255))),
IPTo: utils.IP2Long(strconv.Itoa(rands.Int(0, 255)) + "." + strconv.Itoa(rands.Int(0, 255)) + "." + strconv.Itoa(rands.Int(0, 255)) + "." + strconv.Itoa(rands.Int(0, 255))),
ExpiredAt: 0,
})
}
list.Sort()
t.Log(len(list.itemsMap), "ip")
before := time.Now()
_ = list.Contains(utils.IP2Long("192.168.1.100"))
t.Log(time.Since(before).Seconds()*1000, "ms")
}
@@ -146,6 +209,32 @@ func TestIPList_ContainsAll(t *testing.T) {
}
func TestIPList_ContainsIPStrings(t *testing.T) {
var a = assert.NewAssertion(t)
list := NewIPList()
for i := 0; i < 255; i++ {
list.Add(&IPItem{
Id: int64(i),
IPFrom: utils.IP2Long(strconv.Itoa(i) + ".168.0.1"),
IPTo: utils.IP2Long(strconv.Itoa(i) + ".168.255.1"),
ExpiredAt: 0,
})
}
t.Log(len(list.itemsMap), "ip")
{
item, ok := list.ContainsIPStrings([]string{"192.168.1.100"})
t.Log("item:", item)
a.IsTrue(ok)
}
{
item, ok := list.ContainsIPStrings([]string{"192.167.1.100"})
t.Log("item:", item)
a.IsFalse(ok)
}
}
func TestIPList_Delete(t *testing.T) {
list := NewIPList()
list.Add(&IPItem{
@@ -160,13 +249,13 @@ func TestIPList_Delete(t *testing.T) {
})
t.Log("===BEFORE===")
logs.PrintAsJSON(list.itemsMap, t)
logs.PrintAsJSON(list.ipMap, t)
logs.PrintAsJSON(list.allItemsMap, t)
list.Delete(1)
t.Log("===AFTER===")
logs.PrintAsJSON(list.itemsMap, t)
logs.PrintAsJSON(list.ipMap, t)
logs.PrintAsJSON(list.allItemsMap, t)
}
func TestGC(t *testing.T) {
@@ -184,27 +273,27 @@ func TestGC(t *testing.T) {
ExpiredAt: 0,
})
logs.PrintAsJSON(list.itemsMap, t)
logs.PrintAsJSON(list.ipMap, t)
logs.PrintAsJSON(list.allItemsMap, t)
time.Sleep(2 * time.Second)
t.Log("===AFTER GC===")
logs.PrintAsJSON(list.itemsMap, t)
logs.PrintAsJSON(list.ipMap, t)
logs.PrintAsJSON(list.sortedItems, t)
}
func BenchmarkIPList_Contains(b *testing.B) {
runtime.GOMAXPROCS(1)
list := NewIPList()
for i := 192; i < 194; i++ {
for i := 1; i < 194; i++ {
list.Add(&IPItem{
Id: int64(1),
IPFrom: utils.IP2Long(strconv.Itoa(i) + ".1.0.1"),
IPTo: utils.IP2Long(strconv.Itoa(i) + ".2.0.1"),
Id: int64(i),
IPFrom: utils.IP2Long(strconv.Itoa(i%255) + "." + strconv.Itoa(i%255) + ".0.1"),
IPTo: utils.IP2Long(strconv.Itoa(i%255) + "." + strconv.Itoa(i%255) + ".0.1"),
ExpiredAt: time.Now().Unix() + 60,
})
}
b.Log(len(list.ipMap), "ip")
b.Log(len(list.itemsMap), "ip")
for i := 0; i < b.N; i++ {
_ = list.Contains(utils.IP2Long("192.168.1.100"))
}

View File

@@ -1,12 +1,13 @@
package iplibrary
type LibraryInterface interface {
// 加载数据库文件
// Load 加载数据库文件
Load(dbPath string) error
// 查询IP
// Lookup 查询IP
// 返回结果有可能为空
Lookup(ip string) (*Result, error)
// 关闭数据库文件
// Close 关闭数据库文件
Close()
}

View File

@@ -5,6 +5,7 @@ import (
"github.com/TeaOSLab/EdgeNode/internal/errors"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/lionsoul2014/ip2region/binding/golang/ip2region"
"net"
"strings"
)
@@ -27,6 +28,9 @@ func (this *IP2RegionLibrary) Lookup(ip string) (*Result, error) {
if strings.Contains(ip, ":") {
return nil, nil
}
if net.ParseIP(ip) == nil {
return nil, nil
}
if this.db == nil {
return nil, errors.New("library has not been loaded")

View File

@@ -5,9 +5,9 @@ import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/errors"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/iwind/TeaGo/Tea"
"github.com/iwind/TeaGo/files"
"github.com/iwind/TeaGo/logs"
"github.com/iwind/TeaGo/types"
"regexp"
"strings"
@@ -21,7 +21,7 @@ func init() {
// 初始化
library, err := SharedManager.Load()
if err != nil {
logs.Println("[IP_LIBRARY]" + err.Error())
remotelogs.Error("IP_LIBRARY", err.Error())
return
}
SharedLibrary = library

View File

@@ -20,12 +20,12 @@ import (
var SharedCountryManager = NewCountryManager()
func init() {
events.On(events.EventStart, func() {
events.On(events.EventLoaded, func() {
go SharedCountryManager.Start()
})
}
// 国家信息管理
// CountryManager 国家/地区信息管理
type CountryManager struct {
cacheFile string

View File

@@ -15,12 +15,12 @@ var SharedIPListManager = NewIPListManager()
var IPListUpdateNotify = make(chan bool, 1)
func init() {
events.On(events.EventStart, func() {
events.On(events.EventLoaded, func() {
go SharedIPListManager.Start()
})
}
// IP名单管理
// IPListManager IP名单管理
type IPListManager struct {
// 缓存文件
// 每行一个数据id|from|to|expiredAt
@@ -112,12 +112,16 @@ func (this *IPListManager) fetch() (hasNext bool, err error) {
return false, nil
}
this.locker.Lock()
var changedLists = map[*IPList]bool{}
for _, item := range items {
list, ok := this.listMap[item.ListId]
if !ok {
list = NewIPList()
this.listMap[item.ListId] = list
}
changedLists[list] = true
if item.IsDeleted {
list.Delete(item.Id)
@@ -127,7 +131,7 @@ func (this *IPListManager) fetch() (hasNext bool, err error) {
continue
}
list.Add(&IPItem{
list.AddDelay(&IPItem{
Id: item.Id,
Type: item.Type,
IPFrom: utils.IP2Long(item.IpFrom),
@@ -140,6 +144,11 @@ func (this *IPListManager) fetch() (hasNext bool, err error) {
SharedActionManager.DeleteItem(item.ListType, item)
SharedActionManager.AddItem(item.ListType, item)
}
for changedList := range changedLists {
changedList.Sort()
}
this.locker.Unlock()
this.version = items[len(items)-1].Version

View File

@@ -24,12 +24,12 @@ const (
var SharedProvinceManager = NewProvinceManager()
func init() {
events.On(events.EventStart, func() {
events.On(events.EventLoaded, func() {
go SharedProvinceManager.Start()
})
}
// 国家信息管理
// ProvinceManager 中国省份信息管理
type ProvinceManager struct {
cacheFile string

View File

@@ -7,9 +7,9 @@ import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/errors"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
"github.com/iwind/TeaGo/Tea"
"github.com/iwind/TeaGo/logs"
"os"
"time"
)
@@ -21,16 +21,16 @@ func init() {
})
}
// IP库更新程序
// Updater IP库更新程序
type Updater struct {
}
// 获取新对象
// NewUpdater 获取新对象
func NewUpdater() *Updater {
return &Updater{}
}
// 开始更新
// Start 开始更新
func (this *Updater) Start() {
// 这里不需要太频繁检查更新因为通常不需要更新IP库
ticker := time.NewTicker(1 * time.Hour)
@@ -38,7 +38,7 @@ func (this *Updater) Start() {
for range ticker.C {
err := this.loop()
if err != nil {
logs.Println("[IP_LIBRARY]" + err.Error())
remotelogs.Error("IP_LIBRARY", err.Error())
}
}
}()

50
internal/js/console.go Normal file
View File

@@ -0,0 +1,50 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package js
import (
"encoding/json"
"github.com/iwind/TeaGo/logs"
"reflect"
)
type Console struct {
}
func (this *Console) Log(args ...interface{}) {
for index, arg := range args {
if arg != nil {
switch arg.(type) {
case bool, int, int8, int16, int32, int64, uint, uint8, uint16, uint32, uint64, string:
default:
var argType = reflect.TypeOf(arg)
// 是否有String()方法,如果有直接调用
method, ok := argType.MethodByName("String")
if ok && method.Type.NumIn() == 1 && method.Type.NumOut() == 1 && method.Type.Out(0).Kind() == reflect.String {
args[index] = method.Func.Call([]reflect.Value{reflect.ValueOf(arg)})[0].String()
continue
}
// 转为JSON
argJSON, err := this.toJSON(arg)
if err != nil {
if argType.Kind() == reflect.Func {
args[index] = "[function]"
} else {
args[index] = "[object]"
}
} else {
args[index] = string(argJSON)
}
}
} else {
args[index] = "null"
}
}
logs.Println(append([]interface{}{"[js][console]"}, args...)...)
}
func (this *Console) toJSON(o interface{}) ([]byte, error) {
return json.Marshal(o)
}

View File

@@ -0,0 +1,38 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package js
import (
"testing"
)
func TestConsole_Log(t *testing.T) {
{
vm := NewVM()
_, err := vm.RunString("console.log('Hello', 'world')")
if err != nil {
t.Fatal(err)
}
}
{
vm := NewVM()
_, err := vm.RunString("console.log(null, true, false, 10, 10.123)")
if err != nil {
t.Fatal(err)
}
}
{
vm := NewVM()
_, err := vm.RunString("console.log({ a:1, b:2 })")
if err != nil {
t.Fatal(err)
}
}
{
vm := NewVM()
_, err := vm.RunString("console.log(console.log)")
if err != nil {
t.Fatal(err)
}
}
}

36
internal/js/http.go Normal file
View File

@@ -0,0 +1,36 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package js
type HTTP struct {
r RequestInterface
req *Request
resp *Response
onRequest func(req *Request, resp *Response)
}
func NewHTTP(r RequestInterface) *HTTP {
return &HTTP{
req: NewRequest(r),
resp: NewResponse(r),
}
}
func (this *HTTP) OnRequest(callback func(req *Request, resp *Response)) {
// TODO 考虑是否支持多个callback
this.onRequest = callback
}
func (this *HTTP) OnData(callback func(req *Request, resp *Response)) {
// TODO
}
func (this *HTTP) OnResponse(callback func(req *Request, resp *Response)) {
// TODO
}
func (this *HTTP) TriggerRequest() {
this.onRequest(this.req, this.resp)
}

82
internal/js/request.go Normal file
View File

@@ -0,0 +1,82 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package js
import (
"bytes"
"io/ioutil"
"net"
)
type Request struct {
r RequestInterface
}
func NewRequest(r RequestInterface) *Request {
return &Request{
r: r,
}
}
func (this *Request) Proto() string {
return this.r.JSRequest().Proto
}
func (this *Request) Method() string {
return this.r.JSRequest().Method
}
func (this *Request) Header() map[string][]string {
return this.r.JSRequest().Header
}
func (this *Request) AddHeader(name string, value string) {
this.r.JSRequest().Header[name] = append(this.r.JSRequest().Header[name], value)
}
func (this *Request) SetHeader(name string, value string) {
this.r.JSRequest().Header[name] = []string{value}
}
func (this *Request) RemoteAddr() string {
var remoteAddr = this.r.JSRequest().RemoteAddr
host, _, err := net.SplitHostPort(remoteAddr)
if err == nil {
return host
}
return remoteAddr
}
func (this *Request) Url() *URL {
return NewURL(this.r.JSRequest().URL)
}
func (this *Request) ContentLength() int64 {
return this.r.JSRequest().ContentLength
}
func (this *Request) Body() []byte {
var bodyReader = this.r.JSRequest().Body
if bodyReader == nil {
return []byte{}
}
data, err := ioutil.ReadAll(bodyReader)
if err != nil {
this.r.JSLog("read body failed: " + err.Error())
}
return data
}
func (this *Request) CopyBody() []byte {
var bodyReader = this.r.JSRequest().Body
if bodyReader == nil {
return []byte{}
}
data, err := ioutil.ReadAll(bodyReader)
if err != nil {
this.r.JSLog("read body failed: " + err.Error())
}
this.r.JSRequest().Body = ioutil.NopCloser(bytes.NewReader(data))
return data
}

View File

@@ -0,0 +1,19 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package js
import "net/http"
type RequestInterface interface {
// JSRequest 请求
JSRequest() *http.Request
// JSWriter 响应
JSWriter() http.ResponseWriter
// JSStop 中止请求
JSStop()
// JSLog 打印日志
JSLog(msg ...interface{})
}

124
internal/js/request_test.go Normal file
View File

@@ -0,0 +1,124 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package js_test
import (
"bytes"
"github.com/TeaOSLab/EdgeNode/internal/js"
"github.com/iwind/TeaGo/logs"
"io/ioutil"
"net/http"
"testing"
)
type testRequest struct {
rawRequest *http.Request
rawResponse *testResponse
}
func (this *testRequest) JSRequest() *http.Request {
if this.rawRequest != nil {
return this.rawRequest
}
req, _ := http.NewRequest(http.MethodGet, "https://iwind:123456@goedge.cn/docs?name=Libai&age=20", nil)
req.Header.Set("Server", "edgejs/1.0")
req.Header.Set("Content-Type", "application/json")
req.Body = ioutil.NopCloser(bytes.NewReader([]byte("123456")))
this.rawRequest = req
return req
}
func (this *testRequest) JSWriter() http.ResponseWriter {
if this.rawResponse != nil {
return this.rawResponse
}
this.rawResponse = &testResponse{}
return this.rawResponse
}
func (this *testRequest) JSStop() {
}
func (this *testRequest) JSLog(msg ...interface{}) {
logs.Println(msg...)
}
type testResponse struct {
statusCode int
header http.Header
}
func (this *testResponse) Header() http.Header {
if this.header == nil {
this.header = http.Header{}
}
return this.header
}
func (this *testResponse) Write(p []byte) (int, error) {
return len(p), nil
}
func (this *testResponse) WriteHeader(statusCode int) {
this.statusCode = statusCode
}
func TestRequest(t *testing.T) {
vm := js.NewVM()
vm.SetRequest(&testRequest{})
// 事件监听
_, err := vm.RunString(`
http.onRequest(function (req, resp) {
console.log(req.proto())
let url = req.url()
console.log(url, "port:", url.port(), "args:", url.args())
console.log("username:", url.username(), "password:", url.password())
console.log("uri:", url.uri(), "path:", url.path())
req.addHeader("Server", "1.0")
resp.write("this is response")
console.log(resp)
console.log(req.body())
})
`)
if err != nil {
t.Fatal(err)
}
// 触发事件
_, err = vm.RunString(`http.triggerRequest()`)
if err != nil {
t.Fatal(err)
}
}
func TestRequest_Header(t *testing.T) {
var req = js.NewRequest(&testRequest{})
logs.PrintAsJSON(req.Header(), t)
req.AddHeader("Content-Length", "10")
req.AddHeader("Vary", "1.0")
req.AddHeader("Vary", "2.0")
logs.PrintAsJSON(req.Header(), t)
req.SetHeader("Vary", "3.0")
logs.PrintAsJSON(req.Header(), t)
}
func TestRequest_Body(t *testing.T) {
var req = js.NewRequest(&testRequest{})
t.Log(string(req.Body()))
t.Log(string(req.Body()))
}
func TestRequest_CopyBody(t *testing.T) {
var req = js.NewRequest(&testRequest{})
t.Log(string(req.CopyBody()))
t.Log(string(req.CopyBody()))
}

39
internal/js/response.go Normal file
View File

@@ -0,0 +1,39 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package js
type Response struct {
r RequestInterface
}
func NewResponse(r RequestInterface) *Response {
return &Response{
r: r,
}
}
func (this *Response) Write(s string) error {
_, err := this.r.JSWriter().Write([]byte(s))
return err
}
func (this *Response) Reply(status int) {
this.SetStatus(status)
this.r.JSStop()
}
func (this *Response) Header() map[string][]string {
return this.r.JSWriter().Header()
}
func (this *Response) AddHeader(name string, value string) {
this.r.JSWriter().Header()[name] = append(this.r.JSWriter().Header()[name], value)
}
func (this *Response) SetHeader(name string, value string) {
this.r.JSWriter().Header()[name] = []string{value}
}
func (this *Response) SetStatus(statusCode int) {
this.r.JSWriter().WriteHeader(statusCode)
}

View File

@@ -0,0 +1,16 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package js_test
import (
"github.com/TeaOSLab/EdgeNode/internal/js"
"testing"
)
func TestNewResponse(t *testing.T) {
var resp = js.NewResponse(&testRequest{})
resp.AddHeader("Vary", "1.0")
resp.AddHeader("Vary", "2.0")
resp.SetHeader("Server", "edgejs/1.0")
t.Logf("%#v", resp.Header())
}

90
internal/js/url.go Normal file
View File

@@ -0,0 +1,90 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package js
import (
"github.com/dop251/goja"
"github.com/iwind/TeaGo/types"
"net/url"
)
type URL struct {
u *url.URL
}
func NewURL(u *url.URL) *URL {
return &URL{
u: u,
}
}
func (this *URL) JSNew(args []goja.Value) *URL {
var urlString = ""
if len(args) == 1 {
urlString = args[0].String()
}
u, _ := url.Parse(urlString)
if u == nil {
u = &url.URL{}
}
return NewURL(u)
}
func (this *URL) Port() int {
return types.Int(this.u.Port())
}
func (this *URL) Args() map[string][]string {
return this.u.Query()
}
func (this *URL) Arg(name string) string {
return this.u.Query().Get(name)
}
func (this *URL) Username() string {
if this.u.User != nil {
return this.u.User.Username()
}
return ""
}
func (this *URL) Password() string {
if this.u.User != nil {
password, _ := this.u.User.Password()
return password
}
return ""
}
func (this *URL) Uri() string {
return this.u.RequestURI()
}
func (this *URL) Path() string {
return this.u.Path
}
func (this *URL) Host() string {
return this.u.Host
}
func (this *URL) Fragment() string {
return this.u.Fragment
}
func (this *URL) Hash() string {
if len(this.u.Fragment) > 0 {
return "#" + this.u.Fragment
} else {
return ""
}
}
func (this *URL) Scheme() string {
return this.u.Scheme
}
func (this *URL) String() string {
return this.u.String()
}

18
internal/js/url_test.go Normal file
View File

@@ -0,0 +1,18 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package js
import (
"net/url"
"testing"
)
func TestURL(t *testing.T) {
raw, err := url.Parse("https://iwind:123456@goedge.cn/docs?name=Libai&age=20#a=b")
if err != nil {
t.Fatal(err)
}
var u = NewURL(raw)
t.Log("host:", u.Host())
t.Log("hash:", u.Hash())
}

153
internal/js/vm.go Normal file
View File

@@ -0,0 +1,153 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package js
import (
"errors"
"github.com/dop251/goja"
"github.com/iwind/TeaGo/logs"
"reflect"
"strings"
)
var sharedPrograms []*goja.Program
var sharedConsole = &Console{}
func init() {
// compile programs
}
type VM struct {
vm *goja.Runtime
}
func NewVM() *VM {
vm := goja.New()
vm.SetFieldNameMapper(goja.TagFieldNameMapper("json", true))
// programs
for _, program := range sharedPrograms {
_, _ = vm.RunProgram(program)
}
v := &VM{vm: vm}
v.initVM()
return v
}
func (this *VM) Set(name string, obj interface{}) error {
return this.vm.Set(name, obj)
}
func (this *VM) AddConstructor(name string, instance interface{}) error {
objType := reflect.TypeOf(instance)
if objType.Kind() != reflect.Ptr {
return errors.New("instance should be pointer")
}
// construct
newMethod, ok := objType.MethodByName("JSNew")
if !ok {
return errors.New("can not find 'JSNew()' method in '" + objType.Elem().Name() + "'")
}
var err = this.Set(name, func(call goja.ConstructorCall) *goja.Object {
if newMethod.Type.NumIn() != 2 {
this.throw(errors.New(objType.Elem().Name() + ".JSNew() should accept a '[]goja.Value' argument"))
return nil
}
if newMethod.Type.In(1).String() != "[]goja.Value" {
this.throw(errors.New(objType.Elem().Name() + ".JSNew() should accept a '[]goja.Value' argument"))
return nil
}
// new
var results = newMethod.Func.Call([]reflect.Value{reflect.ValueOf(instance), reflect.ValueOf(call.Arguments)})
if len(results) == 0 {
this.throw(errors.New(objType.Elem().Name() + ".JSNew() should return a valid instance"))
return nil
}
var result = results[0]
if result.Type() != objType {
this.throw(errors.New(objType.Elem().Name() + ".JSNew() should return a same instance"))
return nil
}
// methods
var resultType = result.Type()
var numMethod = result.NumMethod()
for i := 0; i < numMethod; i++ {
var method = resultType.Method(i)
var methodName = strings.ToLower(method.Name[:1]) + method.Name[1:]
err := call.This.Set(methodName, result.MethodByName(method.Name).Interface())
if err != nil {
this.throw(err)
continue
}
}
// 支持属性
var numField = result.Elem().Type().NumField()
for i := 0; i < numField; i++ {
var field = result.Elem().Field(i)
if !field.CanInterface() {
continue
}
var fieldType = objType.Elem().Field(i)
tag, ok := fieldType.Tag.Lookup("json")
if !ok {
tag = fieldType.Name
tag = strings.ToLower(tag[:1]) + tag[1:]
} else {
// TODO 校验tag是否符合变量语法
}
err := call.This.Set(tag, field.Interface())
if err != nil {
this.throw(err)
continue
}
}
return nil
})
return err
}
func (this *VM) RunString(str string) (goja.Value, error) {
defer func() {
e := recover()
if e != nil {
// TODO 需要打印trace
logs.Println("panic:", e)
}
}()
return this.vm.RunString(str)
}
func (this *VM) SetRequest(req RequestInterface) {
{
err := this.vm.Set("http", NewHTTP(req))
if err != nil {
this.throw(err)
}
}
}
func (this *VM) initVM() {
{
err := this.vm.Set("console", sharedConsole)
if err != nil {
this.throw(err)
}
}
}
func (this *VM) throw(err error) {
if err == nil {
return
}
// TODO
logs.Println("js:VM:error: " + err.Error())
}

158
internal/js/vm_test.go Normal file
View File

@@ -0,0 +1,158 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package js
import (
"github.com/dop251/goja"
"testing"
"time"
)
func TestNewVM(t *testing.T) {
before := time.Now()
defer func() {
t.Log(time.Since(before).Seconds()*1000, "ms")
}()
vm := NewVM()
{
v, err := vm.RunString("JSON.stringify({\"a\":\"b\"})")
if err != nil {
t.Fatal(err)
}
t.Log("JSON.stringify():", v)
}
{
v, err := vm.RunString(`JSON.parse('{\"a\":\"b\"}')`)
if err != nil {
t.Fatal(err)
}
t.Log("JSON.parse():", v)
}
{
err := vm.AddConstructor("Url", &URL{})
if err != nil {
t.Fatal("add constructor error:", err)
}
_, err = vm.RunString(`
{
let u = new Url("https://goedge.cn/docs?v=1")
console.log("host:", u.host(), u.uri())
}
{
let u = new Url("https://teaos.cn/downloads?v=1")
console.log("host:", u.host(), u.uri())
}
{
let u = new Url()
console.log("host:", u.host(), u.uri())
}
{
let u = new Url("a", "b", "c")
console.log("host:", u.host(), u.uri())
}
`)
if err != nil {
t.Fatal("add constructor error:" + err.Error())
}
}
}
func TestVM_Program(t *testing.T) {
var s = `
{
let u = new Url("https://goedge.cn/docs?v=1")
//console.log("host:", u.host(), u.uri())
}
{
let u = new Url("https://teaos.cn/downloads?v=1")
//console.log("host:", u.host(), u.uri())
}
{
let u = new Url()
//console.log("host:", u.host(), u.uri())
}
{
let u = new Url("a", "b", "c")
//console.log("host:", u.host(), u.uri())
}
`
program := goja.MustCompile("s", s, true)
before := time.Now()
defer func() {
t.Log(time.Since(before).Seconds()*1000, "ms")
}()
vm := NewVM()
err := vm.AddConstructor("Url", &URL{})
if err != nil {
t.Fatal("add constructor error:", err)
}
//_, err = vm.RunString(s)
_, err = vm.vm.RunProgram(program)
if err != nil {
t.Fatal("add constructor error:" + err.Error())
}
}
func Benchmark_Program(b *testing.B) {
var s = `
{
let u = new Url("https://goedge.cn/docs?v=1")
//console.log("host:", u.host(), u.uri())
}
{
let u = new Url("https://teaos.cn/downloads?v=1")
//console.log("host:", u.host(), u.uri())
}
{
let u = new Url()
//console.log("host:", u.host(), u.uri())
}
{
let u = new Url("a", "b", "c")
//console.log("host:", u.host(), u.uri())
}
{
let u = new Url("https://goedge.cn/docs?v=1")
//console.log("host:", u.host(), u.uri())
}
{
let u = new Url("https://teaos.cn/downloads?v=1")
//console.log("host:", u.host(), u.uri())
}
{
let u = new Url()
//console.log("host:", u.host(), u.uri())
}
{
let u = new Url("a", "b", "c")
//console.log("host:", u.host(), u.uri())
}
`
program := goja.MustCompile("s", s, true)
vm := NewVM()
err := vm.AddConstructor("Url", &URL{})
if err != nil {
b.Fatal("add constructor error:", err)
}
for i := 0; i < b.N; i++ {
//_, err = vm.RunString(s)
_, err = vm.vm.RunProgram(program)
if err != nil {
b.Fatal("add constructor error:" + err.Error())
}
}
}

121
internal/metrics/manager.go Normal file
View File

@@ -0,0 +1,121 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package metrics
import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"strconv"
"sync"
)
var SharedManager = NewManager()
type Manager struct {
tasks map[int64]*Task // itemId => *Task
categoryTasks map[string][]*Task // category => []*Task
locker sync.RWMutex
hasHTTPMetrics bool
hasTCPMetrics bool
hasUDPMetrics bool
}
func NewManager() *Manager {
return &Manager{
tasks: map[int64]*Task{},
categoryTasks: map[string][]*Task{},
}
}
func (this *Manager) Update(items []*serverconfigs.MetricItemConfig) {
this.locker.Lock()
defer this.locker.Unlock()
var newMap = map[int64]*serverconfigs.MetricItemConfig{}
for _, item := range items {
newMap[item.Id] = item
}
// 停用以前的 或 修改现在的
for itemId, task := range this.tasks {
newItem, ok := newMap[itemId]
if !ok || !newItem.IsOn { // 停用以前的
remotelogs.Println("METRIC_MANAGER", "stop task '"+strconv.FormatInt(itemId, 10)+"'")
err := task.Stop()
if err != nil {
remotelogs.Error("METRIC_MANAGER", "stop task '"+strconv.FormatInt(itemId, 10)+"' failed: "+err.Error())
}
delete(this.tasks, itemId)
} else { // 更新已存在的
if newItem.Version != task.item.Version {
remotelogs.Println("METRIC_MANAGER", "update task '"+strconv.FormatInt(itemId, 10)+"'")
task.item = newItem
}
}
}
// 启动新的
for _, newItem := range items {
if !newItem.IsOn {
continue
}
_, ok := this.tasks[newItem.Id]
if !ok {
remotelogs.Println("METRIC_MANAGER", "start task '"+strconv.FormatInt(newItem.Id, 10)+"'")
task := NewTask(newItem)
err := task.Init()
if err != nil {
remotelogs.Error("METRIC_MANAGER", "initialized task failed: "+err.Error())
continue
}
err = task.Start()
if err != nil {
remotelogs.Error("METRIC_MANAGER", "start task failed: "+err.Error())
continue
}
this.tasks[newItem.Id] = task
}
}
// 按分类存放
this.hasHTTPMetrics = false
this.hasTCPMetrics = false
this.hasUDPMetrics = false
this.categoryTasks = map[string][]*Task{}
for _, task := range this.tasks {
tasks := this.categoryTasks[task.item.Category]
tasks = append(tasks, task)
this.categoryTasks[task.item.Category] = tasks
switch task.item.Category {
case serverconfigs.MetricItemCategoryHTTP:
this.hasHTTPMetrics = true
case serverconfigs.MetricItemCategoryTCP:
this.hasTCPMetrics = true
case serverconfigs.MetricItemCategoryUDP:
this.hasUDPMetrics = true
}
}
}
// Add 添加数据
func (this *Manager) Add(obj MetricInterface) {
this.locker.RLock()
for _, task := range this.categoryTasks[obj.MetricCategory()] {
task.Add(obj)
}
this.locker.RUnlock()
}
func (this *Manager) HasHTTPMetrics() bool {
return this.hasHTTPMetrics
}
func (this *Manager) HasTCPMetrics() bool {
return this.hasTCPMetrics
}
func (this *Manager) HasUDPMetrics() bool {
return this.hasUDPMetrics
}

View File

@@ -0,0 +1,63 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package metrics
import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"testing"
)
func TestNewManager(t *testing.T) {
var manager = NewManager()
{
manager.Update([]*serverconfigs.MetricItemConfig{})
for _, task := range manager.tasks {
t.Log(task.item.Id)
}
}
{
t.Log("====")
manager.Update([]*serverconfigs.MetricItemConfig{
{
Id: 1,
},
{
Id: 2,
},
{
Id: 3,
},
})
for _, task := range manager.tasks {
t.Log("task:", task.item.Id)
}
}
{
t.Log("====")
manager.Update([]*serverconfigs.MetricItemConfig{
{
Id: 1,
},
{
Id: 2,
},
})
for _, task := range manager.tasks {
t.Log("task:", task.item.Id)
}
}
{
t.Log("====")
manager.Update([]*serverconfigs.MetricItemConfig{
{
Id: 1,
Version: 1,
},
})
for _, task := range manager.tasks {
t.Log("task:", task.item.Id)
}
}
}

View File

@@ -0,0 +1,17 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package metrics
type MetricInterface interface {
// MetricKey 指标对象
MetricKey(key string) string
// MetricValue 指标值
MetricValue(value string) (result int64, ok bool)
// MetricServerId 服务ID
MetricServerId() int64
// MetricCategory 指标分类
MetricCategory() string
}

22
internal/metrics/stat.go Normal file
View File

@@ -0,0 +1,22 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package metrics
import (
"encoding/json"
"github.com/cespare/xxhash"
"strconv"
)
type Stat struct {
ServerId int64
Keys []string
Hash string
Value int64
Time string
}
func SumStat(serverId int64, keys []string, time string, version int32, itemId int64) string {
keysData, _ := json.Marshal(keys)
return strconv.FormatUint(xxhash.Sum64String(strconv.FormatInt(serverId, 10)+"@"+string(keysData)+"@"+time+"@"+strconv.Itoa(int(version))+"@"+strconv.FormatInt(itemId, 10)), 10)
}

509
internal/metrics/task.go Normal file
View File

@@ -0,0 +1,509 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package metrics
import (
"database/sql"
"encoding/json"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/iwind/TeaGo/Tea"
_ "github.com/mattn/go-sqlite3"
"os"
"strconv"
"strings"
"sync"
"time"
)
const MaxQueueSize = 10240
// Task 单个指标任务
// 数据库存储:
// data/
// metric.$ID.db
// stats
// id, keys, value, time, serverId, hash
// 原理:
// 添加或者有变更时 isUploaded = false
// 上传时检查 isUploaded 状态
// 只上传每个服务中排序最前面的 N 个数据
type Task struct {
item *serverconfigs.MetricItemConfig
isLoaded bool
db *sql.DB
statTableName string
isStopped bool
cleanTicker *utils.Ticker
uploadTicker *utils.Ticker
cleanVersion int32
insertStatStmt *sql.Stmt
deleteByVersionStmt *sql.Stmt
deleteByExpiresTimeStmt *sql.Stmt
selectTopStmt *sql.Stmt
sumStmt *sql.Stmt
serverIdMap map[int64]bool // 所有的服务Ids
timeMap map[string]bool // time => bool
serverIdMapLocker sync.Mutex
statsMap map[string]*Stat
statsLocker sync.Mutex
statsTicker *utils.Ticker
}
// NewTask 获取新任务
func NewTask(item *serverconfigs.MetricItemConfig) *Task {
return &Task{
item: item,
serverIdMap: map[int64]bool{},
timeMap: map[string]bool{},
statsMap: map[string]*Stat{},
}
}
// Init 初始化
func (this *Task) Init() error {
this.statTableName = "stats"
// 检查目录是否存在
var dir = Tea.Root + "/data"
_, err := os.Stat(dir)
if err != nil {
err = os.MkdirAll(dir, 0777)
if err != nil {
return err
}
remotelogs.Println("METRIC", "create data dir '"+dir+"'")
}
db, err := sql.Open("sqlite3", "file:"+dir+"/metric."+strconv.FormatInt(this.item.Id, 10)+".db?cache=shared&mode=rwc&_journal_mode=WAL")
if err != nil {
return err
}
db.SetMaxOpenConns(1)
this.db = db
//创建统计表
_, err = db.Exec(`CREATE TABLE IF NOT EXISTS "` + this.statTableName + `" (
"id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
"hash" varchar(32),
"keys" varchar(1024),
"value" real DEFAULT 0,
"time" varchar(32),
"serverId" integer DEFAULT 0,
"version" integer DEFAULT 0,
"isUploaded" integer DEFAULT 0
);
CREATE INDEX IF NOT EXISTS "serverId"
ON "` + this.statTableName + `" (
"serverId" ASC,
"version" ASC
);
CREATE UNIQUE INDEX IF NOT EXISTS "hash"
ON "` + this.statTableName + `" (
"hash" ASC
);`)
if err != nil {
return err
}
// insert stat stmt
this.insertStatStmt, err = db.Prepare(`INSERT INTO "stats" ("serverId", "hash", "keys", "value", "time", "version", "isUploaded") VALUES (?, ?, ?, ?, ?, ?, 0) ON CONFLICT("hash") DO UPDATE SET "value"="value"+?, "isUploaded"=0`)
if err != nil {
return err
}
// delete by version
this.deleteByVersionStmt, err = db.Prepare(`DELETE FROM "` + this.statTableName + `" WHERE "version"<?`)
if err != nil {
return err
}
// delete by expires time
this.deleteByExpiresTimeStmt, err = db.Prepare(`DELETE FROM "` + this.statTableName + `" WHERE "time"<?`)
if err != nil {
return err
}
// select topN stmt
this.selectTopStmt, err = db.Prepare(`SELECT "id", "hash", "keys", "value", "isUploaded" FROM "` + this.statTableName + `" WHERE "serverId"=? AND "version"=? AND time=? ORDER BY "value" DESC LIMIT 20`)
if err != nil {
return err
}
// sum stmt
this.sumStmt, err = db.Prepare(`SELECT COUNT(*), IFNULL(SUM(value), 0) FROM "` + this.statTableName + `" WHERE "serverId"=? AND "version"=? AND time=?`)
if err != nil {
return err
}
// 所有的服务IDs
err = this.loadServerIdMap()
if err != nil {
return err
}
this.isLoaded = true
return nil
}
// Start 启动任务
func (this *Task) Start() error {
// 读取数据
this.statsTicker = utils.NewTicker(1 * time.Minute)
go func() {
for this.statsTicker.Next() {
this.statsLocker.Lock()
var statsMap = this.statsMap
this.statsMap = map[string]*Stat{}
this.statsLocker.Unlock()
for _, stat := range statsMap {
err := this.InsertStat(stat)
if err != nil {
remotelogs.Error("METRIC", "insert stat failed: "+err.Error())
}
}
}
}()
// 清理
this.cleanTicker = utils.NewTicker(24 * time.Hour)
go func() {
for this.cleanTicker.Next() {
err := this.CleanExpired()
if err != nil {
remotelogs.Error("METRIC", "clean expired stats failed: "+err.Error())
}
}
}()
// 上传
this.uploadTicker = utils.NewTicker(this.item.UploadDuration())
go func() {
for this.uploadTicker.Next() {
err := this.Upload(1 * time.Second)
if err != nil {
remotelogs.Error("METRIC", "upload stats failed: "+err.Error())
}
}
}()
return nil
}
// Add 添加数据
func (this *Task) Add(obj MetricInterface) {
if this.isStopped || !this.isLoaded {
return
}
var keys = []string{}
for _, key := range this.item.Keys {
k := obj.MetricKey(key)
// 忽略499状态
if key == "${status}" && k == "499" {
return
}
keys = append(keys, k)
}
v, ok := obj.MetricValue(this.item.Value)
if !ok {
return
}
var hash = SumStat(obj.MetricServerId(), keys, this.item.CurrentTime(), this.item.Version, this.item.Id)
this.statsLocker.Lock()
oldStat, ok := this.statsMap[hash]
if ok {
oldStat.Value += v
oldStat.Hash = hash
} else {
// 防止过载
if len(this.statsMap) < MaxQueueSize {
this.statsMap[hash] = &Stat{
ServerId: obj.MetricServerId(),
Keys: keys,
Value: v,
Time: this.item.CurrentTime(),
Hash: hash,
}
}
}
this.statsLocker.Unlock()
}
// Stop 停止任务
func (this *Task) Stop() error {
this.isStopped = true
if this.cleanTicker != nil {
this.cleanTicker.Stop()
}
if this.uploadTicker != nil {
this.uploadTicker.Stop()
}
if this.statsTicker != nil {
this.statsTicker.Stop()
}
_ = this.insertStatStmt.Close()
_ = this.deleteByVersionStmt.Close()
_ = this.deleteByExpiresTimeStmt.Close()
_ = this.selectTopStmt.Close()
_ = this.sumStmt.Close()
if this.db != nil {
_ = this.db.Close()
}
return nil
}
// InsertStat 写入数据
func (this *Task) InsertStat(stat *Stat) error {
if this.isStopped {
return nil
}
if stat == nil {
return nil
}
this.serverIdMapLocker.Lock()
this.serverIdMap[stat.ServerId] = true
this.timeMap[stat.Time] = true
this.serverIdMapLocker.Unlock()
keyData, err := json.Marshal(stat.Keys)
if err != nil {
return err
}
_, err = this.insertStatStmt.Exec(stat.ServerId, stat.Hash, keyData, stat.Value, stat.Time, this.item.Version, stat.Value)
if err != nil {
return err
}
return nil
}
// CleanExpired 清理数据
func (this *Task) CleanExpired() error {
if this.isStopped {
return nil
}
// 清除低版本数据
if this.cleanVersion < this.item.Version {
_, err := this.deleteByVersionStmt.Exec(this.item.Version)
if err != nil {
return err
}
this.cleanVersion = this.item.Version
}
// 清除过期的数据
_, err := this.deleteByExpiresTimeStmt.Exec(this.item.LocalExpiresTime())
if err != nil {
return err
}
return nil
}
// Upload 上传数据
func (this *Task) Upload(pauseDuration time.Duration) error {
if this.isStopped {
return nil
}
this.serverIdMapLocker.Lock()
// 服务IDs
var serverIds []int64
for serverId := range this.serverIdMap {
serverIds = append(serverIds, serverId)
}
this.serverIdMap = map[int64]bool{} // 清空数据
// 时间
var times = []string{}
for t := range this.timeMap {
times = append(times, t)
}
this.timeMap = map[string]bool{} // 清空数据
this.serverIdMapLocker.Unlock()
rpcClient, err := rpc.SharedRPC()
if err != nil {
return err
}
for _, serverId := range serverIds {
for _, currentTime := range times {
idStrings, err := func(serverId int64, currentTime string) (ids []string, err error) {
rows, err := this.selectTopStmt.Query(serverId, this.item.Version, currentTime)
if err != nil {
return nil, err
}
var isClosed bool
defer func() {
if isClosed {
return
}
_ = rows.Close()
}()
var pbStats []*pb.UploadingMetricStat
for rows.Next() {
var pbStat = &pb.UploadingMetricStat{}
// "id", "hash", "keys", "value", "isUploaded"
var isUploaded int
var keysData []byte
err = rows.Scan(&pbStat.Id, &pbStat.Hash, &keysData, &pbStat.Value, &isUploaded)
if err != nil {
return nil, err
}
// TODO 先不判断是否已经上传需要改造API进行配合
/**if isUploaded == 1 {
continue
}**/
if len(keysData) > 0 {
err = json.Unmarshal(keysData, &pbStat.Keys)
if err != nil {
return nil, err
}
}
pbStats = append(pbStats, pbStat)
ids = append(ids, strconv.FormatInt(pbStat.Id, 10))
}
// 提前关闭
_ = rows.Close()
isClosed = true
// 上传
if len(pbStats) > 0 {
// 计算总和
count, total, err := this.sum(serverId, currentTime)
if err != nil {
return nil, err
}
_, err = rpcClient.MetricStatRPC().UploadMetricStats(rpcClient.Context(), &pb.UploadMetricStatsRequest{
MetricStats: pbStats,
Time: currentTime,
ServerId: serverId,
ItemId: this.item.Id,
Version: this.item.Version,
Count: count,
Total: float32(total),
})
if err != nil {
return nil, err
}
}
return
}(serverId, currentTime)
if err != nil {
return err
}
if len(idStrings) > 0 {
// 设置为已上传
_, err = this.db.Exec(`UPDATE "` + this.statTableName + `" SET isUploaded=1 WHERE id IN (` + strings.Join(idStrings, ",") + `)`)
if err != nil {
return err
}
}
}
// 休息一下,防止短时间内上传数据过多
if pauseDuration > 0 {
time.Sleep(pauseDuration)
}
}
return nil
}
// 加载服务ID
func (this *Task) loadServerIdMap() error {
{
rows, err := this.db.Query(`SELECT DISTINCT "serverId" FROM `+this.statTableName+" WHERE version=?", this.item.Version)
if err != nil {
return err
}
defer func() {
_ = rows.Close()
}()
var serverId int64
for rows.Next() {
err = rows.Scan(&serverId)
if err != nil {
return err
}
this.serverIdMapLocker.Lock()
this.serverIdMap[serverId] = true
this.serverIdMapLocker.Unlock()
}
}
{
rows, err := this.db.Query(`SELECT DISTINCT "time" FROM `+this.statTableName+" WHERE version=?", this.item.Version)
if err != nil {
return err
}
defer func() {
_ = rows.Close()
}()
var timeString string
for rows.Next() {
err = rows.Scan(&timeString)
if err != nil {
return err
}
this.serverIdMapLocker.Lock()
this.timeMap[timeString] = true
this.serverIdMapLocker.Unlock()
}
}
return nil
}
// 计算数量和综合
func (this *Task) sum(serverId int64, time string) (count int64, total float64, err error) {
rows, err := this.sumStmt.Query(serverId, this.item.Version, time)
if err != nil {
return 0, 0, err
}
defer func() {
_ = rows.Close()
}()
if rows.Next() {
err = rows.Scan(&count, &total)
if err != nil {
return 0, 0, err
}
}
return
}

View File

@@ -0,0 +1,210 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package metrics_test
import (
"fmt"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/metrics"
_ "github.com/iwind/TeaGo/bootstrap"
"github.com/iwind/TeaGo/rands"
"testing"
"time"
)
type testObj struct {
ip string
}
func (this *testObj) MetricKey(key string) string {
return this.ip
}
func (this *testObj) MetricValue(value string) (int64, bool) {
return 1, true
}
func (this *testObj) MetricServerId() int64 {
return int64(rands.Int(1, 100))
}
func (this *testObj) MetricCategory() string {
return "http"
}
func TestTask_Init(t *testing.T) {
var task = metrics.NewTask(&serverconfigs.MetricItemConfig{
Id: 1,
IsOn: false,
Category: "",
Period: 0,
PeriodUnit: "",
Keys: nil,
Value: "",
})
err := task.Init()
if err != nil {
t.Fatal(err)
}
defer func() {
_ = task.Stop()
}()
t.Log("ok")
}
func TestTask_Add(t *testing.T) {
var task = metrics.NewTask(&serverconfigs.MetricItemConfig{
Id: 1,
IsOn: false,
Category: "",
Period: 1,
PeriodUnit: serverconfigs.MetricItemPeriodUnitDay,
Keys: []string{"${remoteAddr}"},
Value: "${countRequest}",
})
err := task.Init()
if err != nil {
t.Fatal(err)
}
err = task.Start()
if err != nil {
t.Fatal(err)
}
defer func() {
_ = task.Stop()
}()
task.Add(&testObj{ip: "127.0.0.2"})
time.Sleep(1 * time.Second) // waiting for inserting
}
func TestTask_Add_Many(t *testing.T) {
var task = metrics.NewTask(&serverconfigs.MetricItemConfig{
Id: 1,
IsOn: false,
Category: "",
Period: 1,
PeriodUnit: serverconfigs.MetricItemPeriodUnitDay,
Keys: []string{"${remoteAddr}"},
Value: "${countRequest}",
Version: 1,
})
err := task.Init()
if err != nil {
t.Fatal(err)
}
err = task.Start()
if err != nil {
t.Fatal(err)
}
defer func() {
_ = task.Stop()
}()
for i := 0; i < 4_000_000; i++ {
task.Add(&testObj{
ip: fmt.Sprintf("%d.%d.%d.%d", rands.Int(0, 255), rands.Int(0, 255), rands.Int(0, 255), rands.Int(0, 255)),
})
if i%10000 == 0 {
time.Sleep(1 * time.Second)
}
}
}
func TestTask_InsertStat(t *testing.T) {
var item = &serverconfigs.MetricItemConfig{
Id: 1,
IsOn: false,
Category: "",
Period: 1,
PeriodUnit: serverconfigs.MetricItemPeriodUnitDay,
Keys: []string{"${remoteAddr}"},
Value: "${countRequest}",
Version: 1,
}
var task = metrics.NewTask(item)
err := task.Init()
if err != nil {
t.Fatal(err)
}
err = task.Start()
if err != nil {
t.Fatal(err)
}
defer func() {
_ = task.Stop()
}()
err = task.InsertStat(&metrics.Stat{
ServerId: 1,
Keys: []string{"127.0.0.1"},
Hash: "",
Value: 1,
Time: item.CurrentTime(),
})
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}
func TestTask_CleanExpired(t *testing.T) {
var task = metrics.NewTask(&serverconfigs.MetricItemConfig{
Id: 1,
IsOn: false,
Category: "",
Period: 1,
PeriodUnit: serverconfigs.MetricItemPeriodUnitDay,
Keys: []string{"${remoteAddr}"},
Value: "${countRequest}",
Version: 1,
})
err := task.Init()
if err != nil {
t.Fatal(err)
}
err = task.Start()
if err != nil {
t.Fatal(err)
}
defer func() {
_ = task.Stop()
}()
err = task.CleanExpired()
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}
func TestTask_Upload(t *testing.T) {
var task = metrics.NewTask(&serverconfigs.MetricItemConfig{
Id: 1,
IsOn: false,
Category: "",
Period: 1,
PeriodUnit: serverconfigs.MetricItemPeriodUnitDay,
Keys: []string{"${remoteAddr}"},
Value: "${countRequest}",
Version: 1,
})
err := task.Init()
if err != nil {
t.Fatal(err)
}
err = task.Start()
if err != nil {
t.Fatal(err)
}
defer func() {
_ = task.Stop()
}()
err = task.Upload(0)
if err != nil {
t.Fatal(err)
}
t.Log("ok")
}

View File

@@ -15,7 +15,7 @@ import (
var SharedValueQueue = NewValueQueue()
func init() {
events.On(events.EventStart, func() {
events.On(events.EventLoaded, func() {
go SharedValueQueue.Start()
})
}
@@ -72,7 +72,8 @@ func (this *ValueQueue) Loop() error {
CreatedAt: value.CreatedAt,
})
if err != nil {
return err
remotelogs.Error("MONITOR", err.Error())
continue
}
}
return nil

View File

@@ -2,6 +2,7 @@ package nodes
import (
"context"
"crypto/tls"
"encoding/json"
"fmt"
"github.com/TeaOSLab/EdgeCommon/pkg/messageconfigs"
@@ -16,6 +17,7 @@ import (
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/iwind/TeaGo/logs"
"io"
"net"
"net/http"
"os/exec"
"strconv"
@@ -143,6 +145,16 @@ func (this *APIStream) handleConnectedAPINode(message *pb.NodeStreamMessage) err
return errors.Wrap(err)
}
remotelogs.Println("API_STREAM", "connected to api node '"+strconv.FormatInt(msg.APINodeId, 10)+"'")
// 重新读取配置
if nodeConfigUpdatedAt == 0 {
select {
case nodeConfigChangedNotify <- true:
default:
}
}
return nil
}
@@ -330,6 +342,15 @@ func (this *APIStream) handlePurgeCache(message *pb.NodeStreamMessage) error {
}()
}
// WEBP缓存
if msg.Type == "file" {
var keys = msg.Keys
for _, key := range keys {
keys = append(keys, key+webpSuffix)
}
msg.Keys = keys
}
err = storage.Purge(msg.Keys, msg.Type)
if err != nil {
this.replyFail(message.RequestId, "purge keys failed: "+err.Error())
@@ -367,7 +388,28 @@ func (this *APIStream) handlePreheatCache(message *pb.NodeStreamMessage) error {
wg := sync.WaitGroup{}
wg.Add(len(msg.Keys))
client := http.Client{} // TODO 可以设置请求超时事件
client := &http.Client{
Timeout: 30 * time.Second, // TODO 可以设置请求超时时间
Transport: &http.Transport{
DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
_, port, err := net.SplitHostPort(addr)
if err != nil {
return nil, err
}
return net.Dial(network, "127.0.0.1:"+port)
},
MaxIdleConns: 4096,
MaxIdleConnsPerHost: 32,
MaxConnsPerHost: 32,
IdleConnTimeout: 2 * time.Minute,
ExpectContinueTimeout: 1 * time.Second,
TLSHandshakeTimeout: 0,
TLSClientConfig: &tls.Config{
InsecureSkipVerify: true,
},
},
}
defer client.CloseIdleConnections()
errorMessages := []string{}
locker := sync.Mutex{}
for _, key := range msg.Keys {
@@ -381,7 +423,9 @@ func (this *APIStream) handlePreheatCache(message *pb.NodeStreamMessage) error {
locker.Unlock()
return
}
// TODO 可以在管理界面自定义Header
req.Header.Set("X-Cache-Action", "preheat")
req.Header.Set("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36")
req.Header.Set("Accept-Encoding", "gzip, deflate, br") // TODO 这里需要记录下缓存是否为gzip的
resp, err := client.Do(req)

View File

@@ -9,14 +9,14 @@ import (
var sharedHTTPAccessLogQueue = NewHTTPAccessLogQueue()
// HTTP访问日志队列
// HTTPAccessLogQueue HTTP访问日志队列
type HTTPAccessLogQueue struct {
queue chan *pb.HTTPAccessLog
}
// 获取新对象
// NewHTTPAccessLogQueue 获取新对象
func NewHTTPAccessLogQueue() *HTTPAccessLogQueue {
// 队列中最大的值,超出此数量的访问日志会被
// 队列中最大的值,超出此数量的访问日志会被
// TODO 需要可以在界面中设置
maxSize := 10000
queue := &HTTPAccessLogQueue{
@@ -27,7 +27,7 @@ func NewHTTPAccessLogQueue() *HTTPAccessLogQueue {
return queue
}
// 开始处理访问日志
// Start 开始处理访问日志
func (this *HTTPAccessLogQueue) Start() {
ticker := time.NewTicker(1 * time.Second)
for range ticker.C {
@@ -38,7 +38,7 @@ func (this *HTTPAccessLogQueue) Start() {
}
}
// 加入新访问日志
// Push 加入新访问日志
func (this *HTTPAccessLogQueue) Push(accessLog *pb.HTTPAccessLog) {
select {
case this.queue <- accessLog:

View File

@@ -7,10 +7,10 @@ import (
"github.com/TeaOSLab/EdgeCommon/pkg/configutils"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/metrics"
"github.com/TeaOSLab/EdgeNode/internal/stats"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/iwind/TeaGo/types"
"golang.org/x/net/http2"
"io"
"net"
"net/http"
@@ -31,6 +31,9 @@ var bytePool1k = utils.NewBytePool(20480, 1024)
var bytePool32k = utils.NewBytePool(20480, 32*1024)
var bytePool128k = utils.NewBytePool(20480, 128*1024)
// errors
var errWritingToClient = errors.New("writing to client error")
// HTTPRequest HTTP请求
type HTTPRequest struct {
// 外部参数
@@ -44,6 +47,7 @@ type HTTPRequest struct {
IsHTTPS bool
// 内部参数
isSubRequest bool
writer *HTTPWriter
web *serverconfigs.HTTPWebConfig // Web配置重要提示由于引用了别的共享的配置所以操作中只能读取不要修改
reverseProxyRef *serverconfigs.ReverseProxyRef // 反向代理引用
@@ -63,12 +67,16 @@ type HTTPRequest struct {
cacheRef *serverconfigs.HTTPCacheRef // 缓存设置
cacheKey string // 缓存使用的Key
isCached bool // 是否已经被缓存
isAttack bool // 是否是攻击请求
bodyData []byte // 读取的Body内容
// WAF相关
firewallPolicyId int64
firewallRuleGroupId int64
firewallRuleSetId int64
firewallRuleId int64
firewallActions []string
tags []string
logAttrs map[string]string
@@ -113,7 +121,7 @@ func (this *HTTPRequest) Do() {
// Web配置
err := this.configureWeb(this.Server.Web, true, 0)
if err != nil {
this.write500(err)
this.write50x(err, http.StatusInternalServerError)
this.doEnd()
return
}
@@ -127,18 +135,24 @@ func (this *HTTPRequest) Do() {
}
// 访问控制
// TODO 需要实现
if !this.isSubRequest && this.web.Auth != nil && this.web.Auth.IsOn {
if this.doAuth() {
this.doEnd()
return
}
}
// 自动跳转到HTTPS
if this.IsHTTP && this.web.RedirectToHttps != nil && this.web.RedirectToHttps.IsOn {
this.doRedirectToHTTPS(this.web.RedirectToHttps)
this.doEnd()
return
if this.doRedirectToHTTPS(this.web.RedirectToHttps) {
this.doEnd()
return
}
}
// Gzip
if this.web.GzipRef != nil && this.web.GzipRef.IsOn && this.web.Gzip != nil && this.web.Gzip.IsOn && this.web.Gzip.Level > 0 {
this.writer.Gzip(this.web.Gzip)
// Compression
if this.web.Compression != nil && this.web.Compression.IsOn && this.web.Compression.Level > 0 {
this.writer.SetCompression(this.web.Compression)
}
// 开始调用
@@ -153,6 +167,16 @@ func (this *HTTPRequest) Do() {
// 开始调用
func (this *HTTPRequest) doBegin() {
// 特殊URL处理
if len(this.rawURI) > 1 && this.rawURI[1] == '.' {
// ACME
// TODO 需要配置是否启用ACME检测
if strings.HasPrefix(this.rawURI, "/.well-known/acme-challenge/") {
this.doACME()
return
}
}
// 统计
if this.web.StatRef != nil && this.web.StatRef.IsOn {
this.doStat()
@@ -165,16 +189,6 @@ func (this *HTTPRequest) doBegin() {
}
}
// 特殊URL处理
if len(this.rawURI) > 1 && this.rawURI[1] == '.' {
// ACME
// TODO 需要配置是否启用ACME检测
if strings.HasPrefix(this.rawURI, "/.well-known/acme-challenge/") {
this.doACME()
return
}
}
// 临时关闭页面
if this.web.Shutdown != nil && this.web.Shutdown.IsOn {
this.doShutdown()
@@ -234,11 +248,20 @@ func (this *HTTPRequest) doEnd() {
// TODO 增加是否开启开关
if this.Server != nil {
if this.isCached {
stats.SharedTrafficStatManager.Add(this.Server.Id, this.writer.sentBodyBytes, this.writer.sentBodyBytes, 1, 1)
stats.SharedTrafficStatManager.Add(this.Server.Id, this.Host, this.writer.sentBodyBytes, this.writer.sentBodyBytes, 1, 1, 0, 0)
} else {
stats.SharedTrafficStatManager.Add(this.Server.Id, this.writer.sentBodyBytes, 0, 1, 0)
if this.isAttack {
stats.SharedTrafficStatManager.Add(this.Server.Id, this.Host, this.writer.sentBodyBytes, 0, 1, 0, 1, this.writer.sentBodyBytes)
} else {
stats.SharedTrafficStatManager.Add(this.Server.Id, this.Host, this.writer.sentBodyBytes, 0, 1, 0, 0, 0)
}
}
}
// 指标
if metrics.SharedManager.HasHTTPMetrics() {
this.doMetricsResponse()
}
}
// RawURI 原始的请求URI
@@ -299,6 +322,11 @@ func (this *HTTPRequest) configureWeb(web *serverconfigs.HTTPWebConfig, isTop bo
this.web.Root = web.Root
}
// remote addr
if web.RemoteAddr != nil && (web.RemoteAddr.IsPrior || isTop) && web.RemoteAddr.IsOn {
this.web.RemoteAddr = web.RemoteAddr
}
// charset
if web.Charset != nil && (web.Charset.IsPrior || isTop) {
this.web.Charset = web.Charset
@@ -310,10 +338,14 @@ func (this *HTTPRequest) configureWeb(web *serverconfigs.HTTPWebConfig, isTop bo
this.web.Websocket = web.Websocket
}
// gzip
if web.GzipRef != nil && (web.GzipRef.IsPrior || isTop) {
this.web.GzipRef = web.GzipRef
this.web.Gzip = web.Gzip
// compression
if web.Compression != nil && (web.Compression.IsPrior || isTop) {
this.web.Compression = web.Compression
}
// webp
if web.WebP != nil && (web.WebP.IsPrior || isTop) {
this.web.WebP = web.WebP
}
// cache
@@ -350,6 +382,11 @@ func (this *HTTPRequest) configureWeb(web *serverconfigs.HTTPWebConfig, isTop bo
this.web.FastcgiList = web.FastcgiList
}
// auth
if web.Auth != nil && (web.Auth.IsPrior || isTop) {
this.web.Auth = web.Auth
}
// 重写规则
if len(web.RewriteRefs) > 0 {
for index, ref := range web.RewriteRefs {
@@ -473,7 +510,9 @@ func (this *HTTPRequest) Format(source string) string {
case "edgeVersion":
return teaconst.Version
case "remoteAddr":
return this.requestRemoteAddr()
return this.requestRemoteAddr(true)
case "remoteAddrValue":
return this.requestRemoteAddr(false)
case "rawRemoteAddr":
addr := this.RawReq.RemoteAddr
host, _, err := net.SplitHostPort(addr)
@@ -487,6 +526,12 @@ func (this *HTTPRequest) Format(source string) string {
return this.requestRemoteUser()
case "requestURI", "requestUri":
return this.rawURI
case "requestURL":
var scheme = "http"
if this.IsHTTPS {
scheme = "https"
}
return scheme + "://" + this.Host + this.rawURI
case "requestPath":
return this.requestPath()
case "requestPathExtension":
@@ -536,6 +581,12 @@ func (this *HTTPRequest) Format(source string) string {
return this.Host
case "referer":
return this.RawReq.Referer()
case "referer.host":
u, err := url.Parse(this.RawReq.Referer())
if err == nil {
return u.Host
}
return ""
case "userAgent":
return this.RawReq.UserAgent()
case "contentType":
@@ -717,22 +768,36 @@ func (this *HTTPRequest) addVarMapping(varMapping map[string]string) {
}
// 获取请求的客户端地址
func (this *HTTPRequest) requestRemoteAddr() string {
func (this *HTTPRequest) requestRemoteAddr(supportVar bool) string {
if supportVar &&
this.web.RemoteAddr != nil &&
this.web.RemoteAddr.IsOn &&
!this.web.RemoteAddr.IsEmpty() {
var remoteAddr = this.Format(this.web.RemoteAddr.Value)
if net.ParseIP(remoteAddr) != nil {
return remoteAddr
}
}
// X-Forwarded-For
forwardedFor := this.RawReq.Header.Get("X-Forwarded-For")
if len(forwardedFor) > 0 {
commaIndex := strings.Index(forwardedFor, ",")
if commaIndex > 0 {
return forwardedFor[:commaIndex]
forwardedFor = forwardedFor[:commaIndex]
}
if net.ParseIP(forwardedFor) != nil {
return forwardedFor
}
return forwardedFor
}
// Real-IP
{
realIP, ok := this.RawReq.Header["X-Real-IP"]
if ok && len(realIP) > 0 {
return realIP[0]
if net.ParseIP(realIP[0]) != nil {
return realIP[0]
}
}
}
@@ -740,7 +805,9 @@ func (this *HTTPRequest) requestRemoteAddr() string {
{
realIP, ok := this.RawReq.Header["X-Real-Ip"]
if ok && len(realIP) > 0 {
return realIP[0]
if net.ParseIP(realIP[0]) != nil {
return realIP[0]
}
}
}
@@ -917,6 +984,11 @@ func (this *HTTPRequest) requestServerPort() int {
return 0
}
// 获取完整的URL
func (this *HTTPRequest) requestFullURL() string {
return this.requestScheme() + "://" + this.Host + this.uri
}
// 设置代理相关头部信息
// 参考https://tools.ietf.org/html/rfc7239
func (this *HTTPRequest) setForwardHeaders(header http.Header) {
@@ -1147,6 +1219,11 @@ func (this *HTTPRequest) canIgnore(err error) bool {
return true
}
// 已读到头
if err == io.EOF || err == io.ErrUnexpectedEOF {
return true
}
// 网络错误
_, ok := err.(*net.OpError)
if ok {
@@ -1154,16 +1231,19 @@ func (this *HTTPRequest) canIgnore(err error) bool {
}
// 客户端主动取消
if err == context.Canceled || err == io.ErrShortWrite {
if err == errWritingToClient || err == context.Canceled || err == io.ErrShortWrite || strings.Contains(err.Error(), "write: connection timed out") || strings.Contains(err.Error(), "write: broken pipe") {
return true
}
// HTTP/2流错误
{
_, ok := err.(http2.StreamError)
if ok {
return true
}
if err.Error() == "http2: stream closed" || err.Error() == "client disconnected" { // errStreamClosed, errClientDisconnected
return true
}
// HTTP内部错误
if strings.HasPrefix(err.Error(), "http:") || strings.HasPrefix(err.Error(), "http2:") {
return true
}
return false
}

View File

@@ -12,6 +12,10 @@ func (this *HTTPRequest) doACME() {
// TODO 对请求进行校验,防止恶意攻击
token := filepath.Base(this.RawReq.URL.Path)
if token == "acme-challenge" || len(token) <= 32 {
this.writer.WriteHeader(http.StatusNotFound)
return
}
rpcClient, err := rpc.SharedRPC()
if err != nil {

View File

@@ -0,0 +1,61 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nodes
import (
"bytes"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"io/ioutil"
"net/http"
)
// 执行认证
func (this *HTTPRequest) doAuth() (shouldStop bool) {
if this.web.Auth == nil || !this.web.Auth.IsOn {
return
}
for _, ref := range this.web.Auth.PolicyRefs {
if !ref.IsOn || ref.AuthPolicy == nil || !ref.AuthPolicy.IsOn {
continue
}
b, err := ref.AuthPolicy.Filter(this.RawReq, func(subReq *http.Request) (status int, err error) {
subReq.TLS = this.RawReq.TLS
subReq.RemoteAddr = this.RawReq.RemoteAddr
subReq.Host = this.RawReq.Host
subReq.Proto = this.RawReq.Proto
subReq.ProtoMinor = this.RawReq.ProtoMinor
subReq.ProtoMajor = this.RawReq.ProtoMajor
subReq.Body = ioutil.NopCloser(bytes.NewReader([]byte{}))
subReq.Header.Set("Referer", this.requestFullURL())
var writer = NewEmptyResponseWriter(this.writer)
this.doSubRequest(writer, subReq)
return writer.StatusCode(), nil
}, this.Format)
if err != nil {
this.write50x(err, http.StatusInternalServerError)
return
}
if b {
return
} else {
if ref.AuthPolicy.Type == serverconfigs.HTTPAuthTypeBasicAuth {
var method = ref.AuthPolicy.Method().(*serverconfigs.HTTPAuthBasicMethod)
var headerValue = "Basic realm=\""
if len(method.Realm) > 0 {
headerValue += method.Realm
} else {
headerValue += this.Host
}
headerValue += "\""
if len(method.Charset) > 0 {
headerValue += ", charset=\"" + method.Charset + "\""
}
this.writer.Header()["WWW-Authenticate"] = []string{headerValue}
}
this.writer.WriteHeader(http.StatusUnauthorized)
return true
}
}
return
}

View File

@@ -5,15 +5,16 @@ import (
"errors"
"github.com/TeaOSLab/EdgeNode/internal/caches"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/iwind/TeaGo/logs"
"net/http"
"path/filepath"
"strconv"
"strings"
"time"
)
// 读取缓存
func (this *HTTPRequest) doCacheRead() (shouldStop bool) {
cachePolicy := sharedNodeConfig.HTTPCachePolicy
cachePolicy := this.Server.HTTPCachePolicy
if cachePolicy == nil || !cachePolicy.IsOn {
return
}
@@ -21,6 +22,12 @@ func (this *HTTPRequest) doCacheRead() (shouldStop bool) {
if this.web.Cache == nil || !this.web.Cache.IsOn || (len(cachePolicy.CacheRefs) == 0 && len(this.web.Cache.CacheRefs) == 0) {
return
}
// 判断是否在预热
if (strings.HasPrefix(this.RawReq.RemoteAddr, "127.") || strings.HasPrefix(this.RawReq.RemoteAddr, "[::1]")) && this.RawReq.Header.Get("X-Cache-Action") == "preheat" {
return
}
var addStatusHeader = this.web.Cache.AddStatusHeader
if addStatusHeader {
defer func() {
@@ -92,6 +99,7 @@ func (this *HTTPRequest) doCacheRead() (shouldStop bool) {
this.cacheRef = nil
return
}
this.cacheKey = key
// 读取缓存
@@ -106,18 +114,32 @@ func (this *HTTPRequest) doCacheRead() (shouldStop bool) {
bytePool32k.Put(buf)
}()
reader, err := storage.OpenReader(key)
if err != nil {
if err == caches.ErrNotFound {
// cache相关变量
this.varMapping["cache.status"] = "MISS"
var reader caches.Reader
var err error
// 是否优先检查WebP
if this.web.WebP != nil &&
this.web.WebP.IsOn &&
this.web.WebP.MatchRequest(filepath.Ext(this.requestPath()), this.Format) &&
this.web.WebP.MatchAccept(this.requestHeader("Accept")) {
reader, _ = storage.OpenReader(key + webpSuffix)
}
// 检查正常的文件
if reader == nil {
reader, err = storage.OpenReader(key)
if err != nil {
if err == caches.ErrNotFound {
// cache相关变量
this.varMapping["cache.status"] = "MISS"
return
}
if !this.canIgnore(err) {
remotelogs.Warn("HTTP_REQUEST_CACHE", "read from cache failed: "+err.Error())
}
return
}
if !this.canIgnore(err) {
remotelogs.Warn("HTTP_REQUEST_CACHE", "read from cache failed: "+err.Error())
}
return
}
defer func() {
_ = reader.Close()
@@ -159,25 +181,22 @@ func (this *HTTPRequest) doCacheRead() (shouldStop bool) {
}
// ETag
// 这里强制设置ETag如果先前源站设置了ETag将会被覆盖避免因为源站的ETag导致源站返回304 Not Modified
var respHeader = this.writer.Header()
var eTag = respHeader.Get("ETag")
var eTag = ""
var lastModifiedAt = reader.LastModified()
if len(eTag) == 0 {
if lastModifiedAt > 0 {
eTag = "\"" + strconv.FormatInt(lastModifiedAt, 10) + "\""
respHeader["ETag"] = []string{eTag}
}
if lastModifiedAt > 0 {
eTag = "\"" + strconv.FormatInt(lastModifiedAt, 10) + "\""
respHeader.Del("Etag")
respHeader["ETag"] = []string{eTag}
}
// 支持 Last-Modified
var modifiedTime = respHeader.Get("Last-Modified")
if len(modifiedTime) == 0 {
if lastModifiedAt > 0 {
modifiedTime = time.Unix(lastModifiedAt, 0).Format("Mon, 02 Jan 2006 15:04:05 GMT")
if len(respHeader.Get("Last-Modified")) == 0 {
respHeader.Set("Last-Modified", modifiedTime)
}
}
// 这里强制设置Last-Modified如果先前源站设置了Last-Modified将会被覆盖避免因为源站的Last-Modified导致源站返回304 Not Modified
var modifiedTime = ""
if lastModifiedAt > 0 {
modifiedTime = time.Unix(lastModifiedAt, 0).Format("Mon, 02 Jan 2006 15:04:05 GMT")
respHeader.Set("Last-Modified", modifiedTime)
}
// 支持 If-None-Match
@@ -185,7 +204,9 @@ func (this *HTTPRequest) doCacheRead() (shouldStop bool) {
// 自定义Header
this.processResponseHeaders(http.StatusNotModified)
this.writer.WriteHeader(http.StatusNotModified)
this.isCached = true
this.cacheRef = nil
this.writer.SetOk()
return true
}
@@ -194,7 +215,9 @@ func (this *HTTPRequest) doCacheRead() (shouldStop bool) {
// 自定义Header
this.processResponseHeaders(http.StatusNotModified)
this.writer.WriteHeader(http.StatusNotModified)
this.isCached = true
this.cacheRef = nil
this.writer.SetOk()
return true
}
@@ -271,7 +294,7 @@ func (this *HTTPRequest) doCacheRead() (shouldStop bool) {
err = reader.ReadBodyRange(buf, rangeSet[0][0], rangeSet[0][1], func(n int) (goNext bool, err error) {
_, err = this.writer.Write(buf[:n])
if err != nil {
return false, err
return false, errWritingToClient
}
return true, nil
})
@@ -301,27 +324,30 @@ func (this *HTTPRequest) doCacheRead() (shouldStop bool) {
_, err = this.writer.WriteString("\r\n--" + boundary + "\r\n")
}
if err != nil {
logs.Error(err)
// 不提示写入客户端错误
return true
}
_, err = this.writer.WriteString("Content-Range: " + "bytes " + strconv.FormatInt(set[0], 10) + "-" + strconv.FormatInt(set[1], 10) + "/" + strconv.FormatInt(reader.BodySize(), 10) + "\r\n")
if err != nil {
logs.Error(err)
// 不提示写入客户端错误
return true
}
if len(contentType) > 0 {
_, err = this.writer.WriteString("Content-Type: " + contentType + "\r\n\r\n")
if err != nil {
logs.Error(err)
// 不提示写入客户端错误
return true
}
}
err := reader.ReadBodyRange(buf, set[0], set[1], func(n int) (goNext bool, err error) {
_, err = this.writer.Write(buf[:n])
return true, err
if err != nil {
return false, errWritingToClient
}
return true, nil
})
if err != nil {
if !this.canIgnore(err) {
@@ -333,16 +359,17 @@ func (this *HTTPRequest) doCacheRead() (shouldStop bool) {
_, err = this.writer.WriteString("\r\n--" + boundary + "--\r\n")
if err != nil {
logs.Error(err)
// 不提示写入客户端错误
return true
}
} else { // 没有Range
this.writer.PrepareCompression(reader.BodySize())
this.writer.WriteHeader(reader.Status())
err = reader.ReadBody(buf, func(n int) (goNext bool, err error) {
_, err = this.writer.Write(buf[:n])
if err != nil {
return false, err
return false, errWritingToClient
}
return true, nil
})
@@ -357,5 +384,8 @@ func (this *HTTPRequest) doCacheRead() (shouldStop bool) {
this.isCached = true
this.cacheRef = nil
this.writer.SetOk()
return true
}

View File

@@ -1,6 +1,7 @@
package nodes
import (
"github.com/iwind/TeaGo/types"
"net/http"
)
@@ -17,30 +18,15 @@ func (this *HTTPRequest) write404() {
_, _ = this.writer.Write([]byte(msg))
}
func (this *HTTPRequest) write500(err error) {
func (this *HTTPRequest) write50x(err error, statusCode int) {
if err != nil {
this.addError(err)
}
statusCode := http.StatusInternalServerError
if this.doPage(statusCode) {
return
}
this.processResponseHeaders(statusCode)
this.writer.WriteHeader(statusCode)
_, _ = this.writer.Write([]byte(http.StatusText(statusCode)))
}
func (this *HTTPRequest) write502(err error) {
if err != nil {
this.addError(err)
}
statusCode := http.StatusBadGateway
if this.doPage(statusCode) {
return
}
this.processResponseHeaders(statusCode)
this.writer.WriteHeader(statusCode)
_, _ = this.writer.Write([]byte("502 Bad Gateway"))
_, _ = this.writer.Write([]byte(types.String(statusCode) + " " + http.StatusText(statusCode)))
}

View File

@@ -15,6 +15,7 @@ import (
"github.com/iwind/gofcgi/pkg/fcgi"
"io"
"net"
"net/http"
"net/url"
"path/filepath"
"strings"
@@ -40,7 +41,7 @@ func (this *HTTPRequest) doFastcgi() (shouldStop bool) {
}
if !env.Has("REMOTE_ADDR") {
env["REMOTE_ADDR"] = this.requestRemoteAddr()
env["REMOTE_ADDR"] = this.requestRemoteAddr(true)
}
if !env.Has("QUERY_STRING") {
u, err := url.ParseRequestURI(this.uri)
@@ -80,7 +81,7 @@ func (this *HTTPRequest) doFastcgi() (shouldStop bool) {
client, err := fcgi.SharedPool(fastcgi.Network(), fastcgi.RealAddress(), uint(poolSize)).Client()
if err != nil {
this.write500(err)
this.write50x(err, http.StatusInternalServerError)
return
}
@@ -158,13 +159,13 @@ func (this *HTTPRequest) doFastcgi() (shouldStop bool) {
resp, stderr, err := client.Call(fcgiReq)
if err != nil {
this.write500(err)
this.write50x(err, http.StatusInternalServerError)
return
}
if len(stderr) > 0 {
err := errors.New("Fastcgi Error: " + strings.TrimSpace(string(stderr)) + " script: " + maps.NewMap(params).GetString("SCRIPT_FILENAME"))
this.write500(err)
this.write50x(err, http.StatusInternalServerError)
return
}

View File

@@ -88,7 +88,7 @@ func (this *HTTPRequest) log() {
RequestId: strconv.FormatInt(this.requestFromTime.UnixNano(), 10) + strconv.FormatInt(atomic.AddInt64(&requestId, 1), 10) + sharedNodeConfig.PaddedId(),
NodeId: sharedNodeConfig.Id,
ServerId: this.Server.Id,
RemoteAddr: this.requestRemoteAddr(),
RemoteAddr: this.requestRemoteAddr(true),
RawRemoteAddr: addr,
RemotePort: int32(this.requestRemotePort()),
RemoteUser: this.requestRemoteUser(),
@@ -128,6 +128,8 @@ func (this *HTTPRequest) log() {
FirewallRuleGroupId: this.firewallRuleGroupId,
FirewallRuleSetId: this.firewallRuleSetId,
FirewallRuleId: this.firewallRuleId,
FirewallActions: this.firewallActions,
Tags: this.tags,
Attrs: this.logAttrs,
}

View File

@@ -0,0 +1,58 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nodes
import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/metrics"
)
// 指标统计 - 响应
// 只需要在结束时调用指标进行统计
func (this *HTTPRequest) doMetricsResponse() {
metrics.SharedManager.Add(this)
}
func (this *HTTPRequest) MetricKey(key string) string {
return this.Format(key)
}
func (this *HTTPRequest) MetricValue(value string) (result int64, ok bool) {
// TODO 需要忽略健康检查的请求,但是同时也要防止攻击者模拟健康检查
switch value {
case "${countRequest}":
return 1, true
case "${countTrafficOut}":
// 这里不包括Header长度
return this.writer.SentBodyBytes(), true
case "${countTrafficIn}":
var hl int64 = 0 // header length
for k, values := range this.RawReq.Header {
for _, v := range values {
hl += int64(len(k) + len(v) + 2 /** k: v **/)
}
}
return this.RawReq.ContentLength + hl, true
case "${countConnection}":
metricNewConnMapLocker.Lock()
_, ok := metricNewConnMap[this.RawReq.RemoteAddr]
if ok {
delete(metricNewConnMap, this.RawReq.RemoteAddr)
}
metricNewConnMapLocker.Unlock()
if ok {
return 1, true
} else {
return 0, false
}
}
return 0, false
}
func (this *HTTPRequest) MetricServerId() int64 {
return this.Server.Id
}
func (this *HTTPRequest) MetricCategory() string {
return serverconfigs.MetricItemCategoryHTTP
}

View File

@@ -1,10 +1,11 @@
package nodes
import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs/shared"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/iwind/TeaGo/Tea"
"github.com/iwind/TeaGo/logs"
"io"
"net/http"
"os"
"regexp"
@@ -20,24 +21,54 @@ func (this *HTTPRequest) doPage(status int) (shouldStop bool) {
for _, page := range this.web.Pages {
if page.Match(status) {
if urlPrefixRegexp.MatchString(page.URL) {
this.doURL(http.MethodGet, page.URL, "", page.NewStatus)
return true
} else {
file := Tea.Root + Tea.DS + page.URL
fp, err := os.Open(file)
if err != nil {
logs.Error(err)
msg := "404 page not found: '" + page.URL + "'"
if len(page.BodyType) == 0 || page.BodyType == shared.BodyTypeURL {
if urlPrefixRegexp.MatchString(page.URL) {
this.doURL(http.MethodGet, page.URL, "", page.NewStatus, true)
return true
} else {
file := Tea.Root + Tea.DS + page.URL
fp, err := os.Open(file)
if err != nil {
logs.Error(err)
msg := "404 page not found: '" + page.URL + "'"
this.writer.WriteHeader(http.StatusNotFound)
_, err := this.writer.Write([]byte(msg))
this.writer.WriteHeader(http.StatusNotFound)
_, err := this.writer.Write([]byte(msg))
if err != nil {
logs.Error(err)
}
return true
}
// 修改状态码
if page.NewStatus > 0 {
// 自定义响应Headers
this.processResponseHeaders(page.NewStatus)
this.writer.WriteHeader(page.NewStatus)
} else {
this.processResponseHeaders(status)
this.writer.WriteHeader(status)
}
buf := bytePool1k.Get()
_, err = utils.CopyWithFilter(this.writer, fp, buf, func(p []byte) []byte {
return []byte(this.Format(string(p)))
})
bytePool1k.Put(buf)
if err != nil {
if !this.canIgnore(err) {
remotelogs.Warn("HTTP_REQUEST_PAGE", "write to client failed: "+err.Error())
}
} else {
this.writer.SetOk()
}
err = fp.Close()
if err != nil {
logs.Error(err)
}
return true
}
return true
} else if page.BodyType == shared.BodyTypeHTML {
// 修改状态码
if page.NewStatus > 0 {
// 自定义响应Headers
@@ -47,9 +78,8 @@ func (this *HTTPRequest) doPage(status int) (shouldStop bool) {
this.processResponseHeaders(status)
this.writer.WriteHeader(status)
}
buf := bytePool1k.Get()
_, err = io.CopyBuffer(this.writer, fp, buf)
bytePool1k.Put(buf)
_, err := this.writer.WriteString(this.Format(page.Body))
if err != nil {
if !this.canIgnore(err) {
remotelogs.Warn("HTTP_REQUEST_PAGE", "write to client failed: "+err.Error())
@@ -57,13 +87,8 @@ func (this *HTTPRequest) doPage(status int) (shouldStop bool) {
} else {
this.writer.SetOk()
}
err = fp.Close()
if err != nil {
logs.Error(err)
}
return true
}
return true
}
}
return false

View File

@@ -7,9 +7,14 @@ import (
"strings"
)
func (this *HTTPRequest) doRedirectToHTTPS(redirectToHTTPSConfig *serverconfigs.HTTPRedirectToHTTPSConfig) {
func (this *HTTPRequest) doRedirectToHTTPS(redirectToHTTPSConfig *serverconfigs.HTTPRedirectToHTTPSConfig) (shouldBreak bool) {
host := this.RawReq.Host
// 检查域名是否匹配
if !redirectToHTTPSConfig.MatchDomain(host) {
return false
}
if len(redirectToHTTPSConfig.Host) > 0 {
if redirectToHTTPSConfig.Port > 0 && redirectToHTTPSConfig.Port != 443 {
host = redirectToHTTPSConfig.Host + ":" + strconv.Itoa(redirectToHTTPSConfig.Port)
@@ -38,4 +43,6 @@ func (this *HTTPRequest) doRedirectToHTTPS(redirectToHTTPSConfig *serverconfigs.
newURL := "https://" + host + this.RawReq.RequestURI
http.Redirect(this.writer, this.RawReq, newURL, statusCode)
return true
}

View File

@@ -8,6 +8,7 @@ import (
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"io"
"net/http"
"net/url"
"strconv"
"strings"
@@ -32,11 +33,15 @@ func (this *HTTPRequest) doReverseProxy() {
// 源站
requestCall := shared.NewRequestCall()
requestCall.Request = this.RawReq
requestCall.Formatter = this.Format
requestCall.Domain = this.Host
origin := this.reverseProxy.NextOrigin(requestCall)
requestCall.CallResponseCallbacks(this.writer)
if origin == nil {
err := errors.New(this.requestPath() + ": no available backends for reverse proxy")
remotelogs.Error("HTTP_REQUEST_REVERSE_PROXY", err.Error())
this.write502(err)
this.write50x(err, http.StatusBadGateway)
return
}
this.origin = origin // 设置全局变量是为了日志等处理
@@ -56,7 +61,7 @@ func (this *HTTPRequest) doReverseProxy() {
if origin.Addr == nil {
err := errors.New(this.requestPath() + ": origin '" + strconv.FormatInt(origin.Id, 10) + "' does not has a address")
remotelogs.Error("HTTP_REQUEST_REVERSE_PROXY", err.Error())
this.write502(err)
this.write50x(err, http.StatusBadGateway)
return
}
this.RawReq.URL.Scheme = origin.Addr.Protocol.Primary().Scheme()
@@ -143,7 +148,7 @@ func (this *HTTPRequest) doReverseProxy() {
client, err := SharedHTTPClientPool.Client(this.RawReq, origin, originAddr)
if err != nil {
remotelogs.Error("HTTP_REQUEST_REVERSE_PROXY", err.Error())
this.write502(err)
this.write50x(err, http.StatusBadGateway)
return
}
@@ -158,10 +163,23 @@ func (this *HTTPRequest) doReverseProxy() {
if err != nil {
// 客户端取消请求,则不提示
httpErr, ok := err.(*url.Error)
if !ok || httpErr.Err != context.Canceled {
// TODO 如果超过最大失败次数,则下线
this.write502(err)
if !ok {
SharedOriginStateManager.Fail(origin, this.reverseProxy, func() {
this.reverseProxy.ResetScheduling()
})
this.write50x(err, http.StatusBadGateway)
remotelogs.Warn("HTTP_REQUEST_REVERSE_PROXY", this.RawReq.URL.String()+"': "+err.Error())
} else if httpErr.Err != context.Canceled {
SharedOriginStateManager.Fail(origin, this.reverseProxy, func() {
this.reverseProxy.ResetScheduling()
})
if httpErr.Timeout() {
this.write50x(err, http.StatusGatewayTimeout)
} else if httpErr.Temporary() {
this.write50x(err, http.StatusServiceUnavailable)
} else {
this.write50x(err, http.StatusBadGateway)
}
remotelogs.Warn("HTTP_REQUEST_REVERSE_PROXY", this.RawReq.URL.String()+"': "+err.Error())
} else {
// 是否为客户端方面的错误
@@ -175,7 +193,7 @@ func (this *HTTPRequest) doReverseProxy() {
}
if !isClientError {
this.write502(err)
this.write50x(err, http.StatusBadGateway)
}
}
if resp != nil && resp.Body != nil {
@@ -183,6 +201,11 @@ func (this *HTTPRequest) doReverseProxy() {
}
return
}
if !origin.IsOk {
SharedOriginStateManager.Success(origin, func() {
this.reverseProxy.ResetScheduling()
})
}
// WAF对出站进行检查
if this.web.FirewallRef != nil && this.web.FirewallRef.IsOn {

View File

@@ -19,7 +19,7 @@ func (this *HTTPRequest) doRewrite() (shouldShop bool) {
if len(this.rewriteRule.ProxyHost) > 0 {
host = this.rewriteRule.ProxyHost
}
this.doURL(this.RawReq.Method, this.rewriteReplace, host, 0)
this.doURL(this.RawReq.Method, this.rewriteReplace, host, 0, false)
return true
}

View File

@@ -109,7 +109,7 @@ func (this *HTTPRequest) doRoot() (isBreak bool) {
}
return
} else {
this.write500(err)
this.write50x(err, http.StatusInternalServerError)
logs.Error(err)
return true
}
@@ -138,7 +138,7 @@ func (this *HTTPRequest) doRoot() (isBreak bool) {
}
return
} else {
this.write500(err)
this.write50x(err, http.StatusInternalServerError)
logs.Error(err)
return true
}
@@ -283,8 +283,7 @@ func (this *HTTPRequest) doRoot() (isBreak bool) {
reader, err := os.OpenFile(filePath, os.O_RDONLY, 0444)
if err != nil {
this.write500(err)
logs.Error(err)
this.write50x(err, http.StatusInternalServerError)
return true
}

View File

@@ -1,10 +1,11 @@
package nodes
import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs/shared"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/iwind/TeaGo/Tea"
"github.com/iwind/TeaGo/logs"
"io"
"net/http"
"os"
)
@@ -17,12 +18,44 @@ func (this *HTTPRequest) doShutdown() {
}
if urlPrefixRegexp.MatchString(shutdown.URL) { // URL
this.doURL(http.MethodGet, shutdown.URL, "", shutdown.Status)
this.doURL(http.MethodGet, shutdown.URL, "", shutdown.Status, true)
return
}
// URL为空则显示文本 TODO 未来可以自定义文本
if len(shutdown.URL) == 0 {
if len(shutdown.BodyType) == 0 || shutdown.BodyType == shared.BodyTypeURL {
// URL为空则显示文本
if len(shutdown.URL) == 0 {
// 自定义响应Headers
if shutdown.Status > 0 {
this.processResponseHeaders(shutdown.Status)
this.writer.WriteHeader(shutdown.Status)
} else {
this.processResponseHeaders(http.StatusOK)
this.writer.WriteHeader(http.StatusOK)
}
_, err := this.writer.WriteString("The site have been shutdown.")
if err != nil {
logs.Error(err)
}
return
}
// 从本地文件中读取
file := Tea.Root + Tea.DS + shutdown.URL
fp, err := os.Open(file)
if err != nil {
logs.Error(err)
msg := "404 page not found: '" + shutdown.URL + "'"
this.writer.WriteHeader(http.StatusNotFound)
_, err = this.writer.Write([]byte(msg))
if err != nil {
logs.Error(err)
}
return
}
// 自定义响应Headers
if shutdown.Status > 0 {
this.processResponseHeaders(shutdown.Status)
@@ -31,51 +64,40 @@ func (this *HTTPRequest) doShutdown() {
this.processResponseHeaders(http.StatusOK)
this.writer.WriteHeader(http.StatusOK)
}
_, err := this.writer.WriteString("The site have been shutdown.")
buf := bytePool1k.Get()
_, err = utils.CopyWithFilter(this.writer, fp, buf, func(p []byte) []byte {
return []byte(this.Format(string(p)))
})
bytePool1k.Put(buf)
if err != nil {
logs.Error(err)
if !this.canIgnore(err) {
remotelogs.Warn("HTTP_REQUEST_SHUTDOWN", "write to client failed: "+err.Error())
}
} else {
this.writer.SetOk()
}
return
}
// 从本地文件中读取
// TODO 支持从数据库中读取文件
file := Tea.Root + Tea.DS + shutdown.URL
fp, err := os.Open(file)
if err != nil {
logs.Error(err)
msg := "404 page not found: '" + shutdown.URL + "'"
this.writer.WriteHeader(http.StatusNotFound)
_, err = this.writer.Write([]byte(msg))
err = fp.Close()
if err != nil {
logs.Error(err)
remotelogs.Warn("HTTP_REQUEST_SHUTDOWN", "close file failed: "+err.Error())
}
return
}
// 自定义响应Headers
if shutdown.Status > 0 {
this.processResponseHeaders(shutdown.Status)
this.writer.WriteHeader(shutdown.Status)
} else {
this.processResponseHeaders(http.StatusOK)
this.writer.WriteHeader(http.StatusOK)
}
buf := bytePool1k.Get()
_, err = io.CopyBuffer(this.writer, fp, buf)
bytePool1k.Put(buf)
if err != nil {
if !this.canIgnore(err) {
remotelogs.Warn("HTTP_REQUEST_SHUTDOWN", "write to client failed: "+err.Error())
} else if shutdown.BodyType == shared.BodyTypeHTML {
// 自定义响应Headers
if shutdown.Status > 0 {
this.processResponseHeaders(shutdown.Status)
this.writer.WriteHeader(shutdown.Status)
} else {
this.processResponseHeaders(http.StatusOK)
this.writer.WriteHeader(http.StatusOK)
}
} else {
this.writer.SetOk()
}
err = fp.Close()
if err != nil {
remotelogs.Warn("HTTP_REQUEST_SHUTDOWN", "close file failed: "+err.Error())
_, err := this.writer.WriteString(this.Format(shutdown.Body))
if err != nil {
if !this.canIgnore(err) {
remotelogs.Warn("HTTP_REQUEST_SHUTDOWN", "write to client failed: "+err.Error())
}
} else {
this.writer.SetOk()
}
}
}

View File

@@ -7,6 +7,8 @@ func (this *HTTPRequest) doStat() {
if this.Server == nil {
return
}
stats.SharedHTTPRequestStatManager.AddRemoteAddr(this.Server.Id, this.requestRemoteAddr())
// 内置的统计
stats.SharedHTTPRequestStatManager.AddRemoteAddr(this.Server.Id, this.requestRemoteAddr(true))
stats.SharedHTTPRequestStatManager.AddUserAgent(this.Server.Id, this.requestHeader("User-Agent"))
}

View File

@@ -0,0 +1,22 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nodes
import "net/http"
// 执行子请求
func (this *HTTPRequest) doSubRequest(writer http.ResponseWriter, rawReq *http.Request) {
// 包装新请求对象
req := &HTTPRequest{
RawReq: rawReq,
RawWriter: writer,
Server: this.Server,
Host: this.Host,
ServerName: this.ServerName,
ServerAddr: this.ServerAddr,
IsHTTP: this.IsHTTP,
IsHTTPS: this.IsHTTPS,
}
req.isSubRequest = true
req.Do()
}

View File

@@ -1,7 +1,6 @@
package nodes
import (
"errors"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/iwind/TeaGo/logs"
@@ -11,7 +10,7 @@ import (
)
// 请求某个URL
func (this *HTTPRequest) doURL(method string, url string, host string, statusCode int) {
func (this *HTTPRequest) doURL(method string, url string, host string, statusCode int, supportVariables bool) {
req, err := http.NewRequest(method, url, this.RawReq.Body)
if err != nil {
logs.Error(err)
@@ -35,8 +34,8 @@ func (this *HTTPRequest) doURL(method string, url string, host string, statusCod
var client = utils.SharedHttpClient(60 * time.Second)
resp, err := client.Do(req)
if err != nil {
logs.Error(errors.New(req.URL.String() + ": " + err.Error()))
this.write500(err)
remotelogs.Error("HTTP_REQUEST_URL", req.URL.String()+": "+err.Error())
this.write50x(err, http.StatusInternalServerError)
return
}
defer func() {
@@ -50,6 +49,9 @@ func (this *HTTPRequest) doURL(method string, url string, host string, statusCod
this.processResponseHeaders(statusCode)
}
if supportVariables {
resp.Header.Del("Content-Length")
}
this.writer.AddHeaders(resp.Header)
if statusCode <= 0 {
this.writer.Prepare(resp.ContentLength, resp.StatusCode)
@@ -67,7 +69,13 @@ func (this *HTTPRequest) doURL(method string, url string, host string, statusCod
// 输出内容
pool := this.bytePool(resp.ContentLength)
buf := pool.Get()
_, err = io.CopyBuffer(this.writer, resp.Body, buf)
if supportVariables {
_, err = utils.CopyWithFilter(this.writer, resp.Body, buf, func(p []byte) []byte {
return []byte(this.Format(string(p)))
})
} else {
_, err = io.CopyBuffer(this.writer, resp.Body, buf)
}
pool.Put(buf)
if err != nil {

View File

@@ -1,6 +1,7 @@
package nodes
import (
"bytes"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs/firewallconfigs"
"github.com/TeaOSLab/EdgeNode/internal/iplibrary"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
@@ -8,11 +9,23 @@ import (
"github.com/TeaOSLab/EdgeNode/internal/waf"
"github.com/iwind/TeaGo/lists"
"github.com/iwind/TeaGo/types"
"io"
"io/ioutil"
"net"
"net/http"
)
// 调用WAF
func (this *HTTPRequest) doWAFRequest() (blocked bool) {
// 当前连接是否已关闭
var conn = this.RawReq.Context().Value(HTTPConnContextKey)
if conn != nil {
trafficConn, ok := conn.(*TrafficConn)
if ok && trafficConn.IsClosed() {
return true
}
}
// 当前服务的独立设置
if this.web.FirewallPolicy != nil && this.web.FirewallPolicy.IsOn {
blocked, breakChecking := this.checkWAFRequest(this.web.FirewallPolicy)
@@ -25,8 +38,8 @@ func (this *HTTPRequest) doWAFRequest() (blocked bool) {
}
// 公用的防火墙设置
if sharedNodeConfig.HTTPFirewallPolicy != nil {
blocked, breakChecking := this.checkWAFRequest(sharedNodeConfig.HTTPFirewallPolicy)
if this.Server.HTTPFirewallPolicy != nil && this.Server.HTTPFirewallPolicy.IsOn {
blocked, breakChecking := this.checkWAFRequest(this.Server.HTTPFirewallPolicy)
if blocked {
return true
}
@@ -40,98 +53,109 @@ func (this *HTTPRequest) doWAFRequest() (blocked bool) {
func (this *HTTPRequest) checkWAFRequest(firewallPolicy *firewallconfigs.HTTPFirewallPolicy) (blocked bool, breakChecking bool) {
// 检查配置是否为空
if firewallPolicy == nil || !firewallPolicy.IsOn || firewallPolicy.Inbound == nil || !firewallPolicy.Inbound.IsOn {
if firewallPolicy == nil || !firewallPolicy.IsOn || firewallPolicy.Inbound == nil || !firewallPolicy.Inbound.IsOn || firewallPolicy.Mode == firewallconfigs.FirewallModeBypass {
return
}
// 检查IP白名单
remoteAddrs := this.requestRemoteAddrs()
inbound := firewallPolicy.Inbound
if inbound.AllowListRef != nil && inbound.AllowListRef.IsOn && inbound.AllowListRef.ListId > 0 {
list := iplibrary.SharedIPListManager.FindList(inbound.AllowListRef.ListId)
if list != nil {
found, _ := list.ContainsIPStrings(remoteAddrs)
if found {
breakChecking = true
return
if inbound == nil {
return
}
for _, ref := range inbound.AllAllowListRefs() {
if ref.IsOn && ref.ListId > 0 {
list := iplibrary.SharedIPListManager.FindList(ref.ListId)
if list != nil {
_, found := list.ContainsIPStrings(remoteAddrs)
if found {
breakChecking = true
return
}
}
}
}
// 检查IP黑名单
if inbound.DenyListRef != nil && inbound.DenyListRef.IsOn && inbound.DenyListRef.ListId > 0 {
list := iplibrary.SharedIPListManager.FindList(inbound.DenyListRef.ListId)
if list != nil {
found, item := list.ContainsIPStrings(remoteAddrs)
if found {
// 触发事件
if item != nil && len(item.EventLevel) > 0 {
actions := iplibrary.SharedActionManager.FindEventActions(item.EventLevel)
for _, action := range actions {
goNext, err := action.DoHTTP(this.RawReq, this.RawWriter)
if err != nil {
remotelogs.Error("HTTP_REQUEST_WAF", "do action '"+err.Error()+"' failed: "+err.Error())
return true, false
}
if !goNext {
this.disableLog = true
return true, false
if firewallPolicy.Mode == firewallconfigs.FirewallModeDefend {
for _, ref := range inbound.AllDenyListRefs() {
if ref.IsOn && ref.ListId > 0 {
list := iplibrary.SharedIPListManager.FindList(ref.ListId)
if list != nil {
item, found := list.ContainsIPStrings(remoteAddrs)
if found {
// 触发事件
if item != nil && len(item.EventLevel) > 0 {
actions := iplibrary.SharedActionManager.FindEventActions(item.EventLevel)
for _, action := range actions {
goNext, err := action.DoHTTP(this.RawReq, this.RawWriter)
if err != nil {
remotelogs.Error("HTTP_REQUEST_WAF", "do action '"+err.Error()+"' failed: "+err.Error())
return true, false
}
if !goNext {
this.disableLog = true
return true, false
}
}
}
// TODO 需要记录日志信息
this.writer.WriteHeader(http.StatusForbidden)
this.writer.Close()
// 停止日志
this.disableLog = true
return true, false
}
}
// TODO 需要记录日志信息
this.writer.WriteHeader(http.StatusForbidden)
this.writer.Close()
// 停止日志
this.disableLog = true
return true, false
}
}
}
// 检查地区封禁
if iplibrary.SharedLibrary != nil {
if firewallPolicy.Inbound.Region != nil && firewallPolicy.Inbound.Region.IsOn {
regionConfig := firewallPolicy.Inbound.Region
if regionConfig.IsNotEmpty() {
for _, remoteAddr := range remoteAddrs {
result, err := iplibrary.SharedLibrary.Lookup(remoteAddr)
if err != nil {
remotelogs.Error("HTTP_REQUEST_WAF", "iplibrary lookup failed: "+err.Error())
} else if result != nil {
// 检查国家级别封禁
if len(regionConfig.DenyCountryIds) > 0 && len(result.Country) > 0 {
countryId := iplibrary.SharedCountryManager.Lookup(result.Country)
if countryId > 0 && lists.ContainsInt64(regionConfig.DenyCountryIds, countryId) {
// TODO 可以配置对封禁的处理方式等
// TODO 需要记录日志信息
this.writer.WriteHeader(http.StatusForbidden)
this.writer.Close()
if firewallPolicy.Mode == firewallconfigs.FirewallModeDefend {
if iplibrary.SharedLibrary != nil {
if firewallPolicy.Inbound.Region != nil && firewallPolicy.Inbound.Region.IsOn {
regionConfig := firewallPolicy.Inbound.Region
if regionConfig.IsNotEmpty() {
for _, remoteAddr := range remoteAddrs {
result, err := iplibrary.SharedLibrary.Lookup(remoteAddr)
if err != nil {
remotelogs.Error("HTTP_REQUEST_WAF", "iplibrary lookup failed: "+err.Error())
} else if result != nil {
// 检查国家级别封禁
if len(regionConfig.DenyCountryIds) > 0 && len(result.Country) > 0 {
countryId := iplibrary.SharedCountryManager.Lookup(result.Country)
if countryId > 0 && lists.ContainsInt64(regionConfig.DenyCountryIds, countryId) {
// TODO 可以配置对封禁的处理方式等
// TODO 需要记录日志信息
this.writer.WriteHeader(http.StatusForbidden)
this.writer.Close()
// 停止日志
this.disableLog = true
// 停止日志
this.disableLog = true
return true, false
return true, false
}
}
}
// 检查省份封禁
if len(regionConfig.DenyProvinceIds) > 0 && len(result.Province) > 0 {
provinceId := iplibrary.SharedProvinceManager.Lookup(result.Province)
if provinceId > 0 && lists.ContainsInt64(regionConfig.DenyProvinceIds, provinceId) {
// TODO 可以配置对封禁的处理方式等
// TODO 需要记录日志信息
this.writer.WriteHeader(http.StatusForbidden)
this.writer.Close()
// 检查省份封禁
if len(regionConfig.DenyProvinceIds) > 0 && len(result.Province) > 0 {
provinceId := iplibrary.SharedProvinceManager.Lookup(result.Province)
if provinceId > 0 && lists.ContainsInt64(regionConfig.DenyProvinceIds, provinceId) {
// TODO 可以配置对封禁的处理方式等
// TODO 需要记录日志信息
this.writer.WriteHeader(http.StatusForbidden)
this.writer.Close()
// 停止日志
this.disableLog = true
// 停止日志
this.disableLog = true
return true, false
return true, false
}
}
}
}
@@ -145,23 +169,36 @@ func (this *HTTPRequest) checkWAFRequest(firewallPolicy *firewallconfigs.HTTPFir
if w == nil {
return
}
goNext, ruleGroup, ruleSet, err := w.MatchRequest(this.RawReq, this.writer)
w.OnAction(func(action waf.ActionInterface) (goNext bool) {
switch action.Code() {
case waf.ActionTag:
this.tags = action.(*waf.TagAction).Tags
}
return true
})
goNext, ruleGroup, ruleSet, err := w.MatchRequest(this, this.writer)
if err != nil {
remotelogs.Error("HTTP_REQUEST_WAF", this.rawURI+": "+err.Error())
return
}
if ruleSet != nil {
if ruleSet.Action != waf.ActionAllow {
if ruleSet.HasSpecialActions() {
this.firewallPolicyId = firewallPolicy.Id
this.firewallRuleGroupId = types.Int64(ruleGroup.Id)
this.firewallRuleSetId = types.Int64(ruleSet.Id)
if ruleSet.HasAttackActions() {
this.isAttack = true
}
// 添加统计
stats.SharedHTTPRequestStatManager.AddFirewallRuleGroupId(this.Server.Id, this.firewallRuleGroupId, ruleSet.Action)
stats.SharedHTTPRequestStatManager.AddFirewallRuleGroupId(this.Server.Id, this.firewallRuleGroupId, ruleSet.Actions)
}
this.logAttrs["waf.action"] = ruleSet.Action
this.firewallActions = append(ruleSet.ActionCodes(), firewallPolicy.Mode)
}
return !goNext, false
@@ -169,8 +206,26 @@ func (this *HTTPRequest) checkWAFRequest(firewallPolicy *firewallconfigs.HTTPFir
// call response waf
func (this *HTTPRequest) doWAFResponse(resp *http.Response) (blocked bool) {
firewallPolicy := sharedNodeConfig.HTTPFirewallPolicy
if firewallPolicy == nil || !firewallPolicy.IsOn || !firewallPolicy.Outbound.IsOn {
// 当前服务的独立设置
if this.web.FirewallPolicy != nil && this.web.FirewallPolicy.IsOn {
blocked := this.checkWAFResponse(this.web.FirewallPolicy, resp)
if blocked {
return true
}
}
// 公用的防火墙设置
if this.Server.HTTPFirewallPolicy != nil && this.Server.HTTPFirewallPolicy.IsOn {
blocked := this.checkWAFResponse(this.Server.HTTPFirewallPolicy, resp)
if blocked {
return true
}
}
return
}
func (this *HTTPRequest) checkWAFResponse(firewallPolicy *firewallconfigs.HTTPFirewallPolicy, resp *http.Response) (blocked bool) {
if firewallPolicy == nil || !firewallPolicy.IsOn || !firewallPolicy.Outbound.IsOn || firewallPolicy.Mode == firewallconfigs.FirewallModeBypass {
return
}
@@ -179,24 +234,93 @@ func (this *HTTPRequest) doWAFResponse(resp *http.Response) (blocked bool) {
return
}
goNext, ruleGroup, ruleSet, err := w.MatchResponse(this.RawReq, resp, this.writer)
w.OnAction(func(action waf.ActionInterface) (goNext bool) {
switch action.Code() {
case waf.ActionTag:
this.tags = action.(*waf.TagAction).Tags
}
return true
})
goNext, ruleGroup, ruleSet, err := w.MatchResponse(this, resp, this.writer)
if err != nil {
remotelogs.Error("HTTP_REQUEST_WAF", this.rawURI+": "+err.Error())
return
}
if ruleSet != nil {
if ruleSet.Action != waf.ActionAllow {
if ruleSet.HasSpecialActions() {
this.firewallPolicyId = firewallPolicy.Id
this.firewallRuleGroupId = types.Int64(ruleGroup.Id)
this.firewallRuleSetId = types.Int64(ruleSet.Id)
if ruleSet.HasAttackActions() {
this.isAttack = true
}
// 添加统计
stats.SharedHTTPRequestStatManager.AddFirewallRuleGroupId(this.Server.Id, this.firewallRuleGroupId, ruleSet.Action)
stats.SharedHTTPRequestStatManager.AddFirewallRuleGroupId(this.Server.Id, this.firewallRuleGroupId, ruleSet.Actions)
}
this.logAttrs["waf.action"] = ruleSet.Action
this.firewallActions = append(ruleSet.ActionCodes(), firewallPolicy.Mode)
}
return !goNext
}
// WAFRaw 原始请求
func (this *HTTPRequest) WAFRaw() *http.Request {
return this.RawReq
}
// WAFRemoteIP 客户端IP
func (this *HTTPRequest) WAFRemoteIP() string {
return this.requestRemoteAddr(true)
}
// WAFGetCacheBody 获取缓存中的Body
func (this *HTTPRequest) WAFGetCacheBody() []byte {
return this.bodyData
}
// WAFSetCacheBody 设置Body
func (this *HTTPRequest) WAFSetCacheBody(body []byte) {
this.bodyData = body
}
// WAFReadBody 读取Body
func (this *HTTPRequest) WAFReadBody(max int64) (data []byte, err error) {
if this.RawReq.ContentLength > 0 {
data, err = ioutil.ReadAll(io.LimitReader(this.RawReq.Body, max))
}
return
}
// WAFRestoreBody 恢复Body
func (this *HTTPRequest) WAFRestoreBody(data []byte) {
if len(data) > 0 {
rawReader := bytes.NewBuffer(data)
buf := make([]byte, 1024)
_, _ = io.CopyBuffer(rawReader, this.RawReq.Body, buf)
this.RawReq.Body = ioutil.NopCloser(rawReader)
}
}
// WAFServerId 服务ID
func (this *HTTPRequest) WAFServerId() int64 {
return this.Server.Id
}
// WAFClose 关闭连接
func (this *HTTPRequest) WAFClose() {
requestConn := this.RawReq.Context().Value(HTTPConnContextKey)
if requestConn == nil {
return
}
conn, ok := requestConn.(net.Conn)
if ok {
_ = conn.Close()
return
}
return
}

View File

@@ -2,7 +2,6 @@ package nodes
import (
"errors"
"github.com/iwind/TeaGo/logs"
"io"
"net/http"
"net/url"
@@ -43,8 +42,7 @@ func (this *HTTPRequest) doWebsocket() {
// TODO 增加N次错误重试重试的时候需要尝试不同的源站
originConn, err := OriginConnect(this.origin, this.RawReq.RemoteAddr)
if err != nil {
logs.Error(err)
this.write500(err)
this.write50x(err, http.StatusBadGateway)
return
}
defer func() {
@@ -53,15 +51,13 @@ func (this *HTTPRequest) doWebsocket() {
err = this.RawReq.Write(originConn)
if err != nil {
logs.Error(err)
this.write500(err)
this.write50x(err, http.StatusBadGateway)
return
}
clientConn, _, err := this.writer.Hijack()
if err != nil {
logs.Error(err)
this.write500(err)
this.write50x(err, http.StatusInternalServerError)
return
}
defer func() {

View File

@@ -3,32 +3,57 @@ package nodes
import (
"bufio"
"bytes"
"compress/gzip"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/caches"
"github.com/TeaOSLab/EdgeNode/internal/compressions"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/utils"
"github.com/chai2010/webp"
"github.com/iwind/TeaGo/lists"
"github.com/iwind/TeaGo/types"
_ "golang.org/x/image/bmp"
_ "golang.org/x/image/webp"
"image"
_ "image/gif"
_ "image/jpeg"
_ "image/png"
"io"
"net"
"net/http"
"path/filepath"
"strings"
"sync/atomic"
)
// 限制WebP能够同时使用的Buffer内存使用量
const webpMaxBufferSize int64 = 1_000_000_000
const webpSuffix = "@GOEDGE_WEBP"
var webpTotalBufferSize int64 = 0
var webpBufferPool = utils.NewBufferPool(1024)
// HTTPWriter 响应Writer
type HTTPWriter struct {
req *HTTPRequest
writer http.ResponseWriter
gzipConfig *serverconfigs.HTTPGzipConfig
gzipWriter *gzip.Writer
size int64
webpIsEncoding bool
webpBuffer *bytes.Buffer
webpIsWriting bool
compressionConfig *serverconfigs.HTTPCompressionConfig
compressionWriter compressions.Writer
compressionType serverconfigs.HTTPCompressionType
statusCode int
sentBodyBytes int64
bodyCopying bool
body []byte
gzipBodyBuffer *bytes.Buffer // 当使用gzip压缩时使用
gzipBodyWriter *gzip.Writer // 当使用gzip压缩时使用
bodyCopying bool
body []byte
compressionBodyBuffer *bytes.Buffer // 当使用压缩时使用
compressionBodyWriter compressions.Writer // 当使用压缩时使用
cacheWriter caches.Writer // 缓存写入
cacheStorage caches.StorageInterface
@@ -48,29 +73,39 @@ func NewHTTPWriter(req *HTTPRequest, httpResponseWriter http.ResponseWriter) *HT
func (this *HTTPWriter) Reset(httpResponseWriter http.ResponseWriter) {
this.writer = httpResponseWriter
this.gzipConfig = nil
this.gzipWriter = nil
this.compressionConfig = nil
this.compressionWriter = nil
this.statusCode = 0
this.sentBodyBytes = 0
this.bodyCopying = false
this.body = nil
this.gzipBodyBuffer = nil
this.gzipBodyWriter = nil
this.compressionBodyBuffer = nil
this.compressionBodyWriter = nil
}
// Gzip 设置Gzip
func (this *HTTPWriter) Gzip(config *serverconfigs.HTTPGzipConfig) {
this.gzipConfig = config
// SetCompression 设置内容压缩配置
func (this *HTTPWriter) SetCompression(config *serverconfigs.HTTPCompressionConfig) {
this.compressionConfig = config
}
// Prepare 准备输出
// 缓存不调用此函数
func (this *HTTPWriter) Prepare(size int64, status int) {
this.size = size
this.statusCode = status
this.prepareGzip(size)
if status == http.StatusOK {
this.prepareWebP(size)
}
this.prepareCache(size)
// 在WebP模式下压缩暂不可用
if !this.webpIsEncoding {
this.PrepareCompression(size)
}
}
// Raw 包装前的原始的Writer
@@ -103,40 +138,46 @@ func (this *HTTPWriter) AddHeaders(header http.Header) {
// Write 写入数据
func (this *HTTPWriter) Write(data []byte) (n int, err error) {
if this.writer != nil {
if this.gzipWriter != nil {
n, err = this.gzipWriter.Write(data)
} else {
n, err = this.writer.Write(data)
}
if n > 0 {
this.sentBodyBytes += int64(n)
}
n = len(data)
// 写入缓存
if this.cacheWriter != nil {
_, err = this.cacheWriter.Write(data)
if err != nil {
_ = this.cacheWriter.Discard()
this.cacheWriter = nil
remotelogs.Error("HTTP_WRITER", "write cache failed: "+err.Error())
}
}
} else {
if n == 0 {
n = len(data) // 防止出现short write错误
}
}
if this.bodyCopying {
if this.gzipBodyWriter != nil {
_, err := this.gzipBodyWriter.Write(data)
if err != nil {
remotelogs.Error("HTTP_WRITER", err.Error())
}
if this.writer != nil {
if this.webpIsEncoding && !this.webpIsWriting {
this.webpBuffer.Write(data)
} else {
this.body = append(this.body, data...)
// 写入压缩
var n1 int
if this.compressionWriter != nil {
n1, err = this.compressionWriter.Write(data)
} else {
n1, err = this.writer.Write(data)
}
if n1 > 0 {
this.sentBodyBytes += int64(n1)
}
// 写入缓存
if this.cacheWriter != nil {
_, err = this.cacheWriter.Write(data)
if err != nil {
_ = this.cacheWriter.Discard()
this.cacheWriter = nil
remotelogs.Error("HTTP_WRITER", "write cache failed: "+err.Error())
}
}
if this.bodyCopying {
if this.compressionBodyWriter != nil {
_, err := this.compressionBodyWriter.Write(data)
if err != nil {
remotelogs.Error("HTTP_WRITER", err.Error())
}
} else {
this.body = append(this.body, data...)
}
}
}
}
return
}
@@ -210,30 +251,92 @@ func (this *HTTPWriter) SetOk() {
// Close 关闭
func (this *HTTPWriter) Close() {
// gzip writer
if this.gzipWriter != nil {
if this.bodyCopying && this.gzipBodyWriter != nil {
_ = this.gzipBodyWriter.Close()
this.body = this.gzipBodyBuffer.Bytes()
if this.webpIsEncoding {
defer func() {
atomic.AddInt64(&webpTotalBufferSize, -this.size*32)
webpBufferPool.Put(this.webpBuffer)
}()
}
// webp writer
if this.isOk && this.webpIsEncoding {
var bufferLen = int64(this.webpBuffer.Len())
atomic.AddInt64(&webpTotalBufferSize, bufferLen*8)
imageData, _, err := image.Decode(this.webpBuffer)
if err != nil {
_, _ = io.Copy(this.writer, this.webpBuffer)
// 处理缓存
if this.cacheWriter != nil {
_ = this.cacheWriter.Discard()
}
this.cacheWriter = nil
} else {
var f = types.Float32(this.req.web.WebP.Quality)
if f > 100 {
f = 100
}
this.webpIsWriting = true
err = webp.Encode(this, imageData, &webp.Options{
Lossless: false,
Quality: f,
Exact: true,
})
if err != nil {
if !this.req.canIgnore(err) {
remotelogs.Error("HTTP_WRITER", "encode webp failed: "+err.Error())
}
// 处理缓存
if this.cacheWriter != nil {
_ = this.cacheWriter.Discard()
}
this.cacheWriter = nil
}
}
_ = this.gzipWriter.Close()
this.gzipWriter = nil
atomic.AddInt64(&webpTotalBufferSize, -bufferLen*8)
this.webpBuffer.Reset()
}
// compression writer
if this.compressionWriter != nil {
if this.bodyCopying && this.compressionBodyWriter != nil {
_ = this.compressionBodyWriter.Close()
this.body = this.compressionBodyBuffer.Bytes()
}
_ = this.compressionWriter.Close()
this.compressionWriter = nil
}
// cache writer
if this.cacheWriter != nil {
if this.isOk {
err := this.cacheWriter.Close()
if err == nil {
this.cacheStorage.AddToList(&caches.Item{
Type: this.cacheWriter.ItemType(),
Key: this.cacheWriter.Key(),
ExpiredAt: this.cacheWriter.ExpiredAt(),
HeaderSize: this.cacheWriter.HeaderSize(),
BodySize: this.cacheWriter.BodySize(),
Host: this.req.Host,
ServerId: this.req.Server.Id,
})
// 对比Content-Length
contentLengthString := this.Header().Get("Content-Length")
if len(contentLengthString) > 0 {
contentLength := types.Int64(contentLengthString)
if contentLength != this.cacheWriter.BodySize() {
this.isOk = false
_ = this.cacheWriter.Discard()
}
}
if this.isOk {
err := this.cacheWriter.Close()
if err == nil {
this.cacheStorage.AddToList(&caches.Item{
Type: this.cacheWriter.ItemType(),
Key: this.cacheWriter.Key(),
ExpiredAt: this.cacheWriter.ExpiredAt(),
HeaderSize: this.cacheWriter.HeaderSize(),
BodySize: this.cacheWriter.BodySize(),
Host: this.req.Host,
ServerId: this.req.Server.Id,
})
}
}
} else {
_ = this.cacheWriter.Discard()
@@ -258,35 +361,29 @@ func (this *HTTPWriter) Flush() {
}
}
// 准备Gzip
func (this *HTTPWriter) prepareGzip(size int64) {
if this.gzipConfig == nil || this.gzipConfig.Level <= 0 {
return
}
// 准备Webp
func (this *HTTPWriter) prepareWebP(size int64) {
if this.req.web != nil &&
this.req.web.WebP != nil &&
this.req.web.WebP.IsOn &&
this.req.web.WebP.MatchResponse(this.Header().Get("Content-Type"), size, filepath.Ext(this.req.requestPath()), this.req.Format) &&
this.req.web.WebP.MatchAccept(this.req.requestHeader("Accept")) &&
len(this.writer.Header().Get("Content-Encoding")) == 0 &&
atomic.LoadInt64(&webpTotalBufferSize) < webpMaxBufferSize {
this.webpIsEncoding = true
this.webpBuffer = webpBufferPool.Get()
// 判断Accept是否支持gzip
if !strings.Contains(this.req.requestHeader("Accept-Encoding"), "gzip") {
return
}
this.Header().Del("Content-Length")
this.Header().Set("Content-Type", "image/webp")
// 尺寸和类型
if size < this.gzipConfig.MinBytes() || (this.gzipConfig.MaxBytes() > 0 && size > this.gzipConfig.MaxBytes()) {
return
atomic.AddInt64(&webpTotalBufferSize, size*32)
}
}
// 校验其他条件
if this.gzipConfig.Conds != nil {
if len(this.gzipConfig.Conds.Groups) > 0 {
if !this.gzipConfig.Conds.MatchRequest(this.req.Format) || !this.gzipConfig.Conds.MatchResponse(this.req.Format) {
return
}
} else {
// 默认校验文档类型
contentType := this.writer.Header().Get("Content-Type")
if len(contentType) > 0 && (!strings.HasPrefix(contentType, "text/") && !strings.HasPrefix(contentType, "application/")) {
return
}
}
// PrepareCompression 准备压缩
func (this *HTTPWriter) PrepareCompression(size int64) {
if this.compressionConfig == nil || !this.compressionConfig.IsOn || this.compressionConfig.Level <= 0 {
return
}
// 如果已经有编码则不处理
@@ -294,9 +391,21 @@ func (this *HTTPWriter) prepareGzip(size int64) {
return
}
// gzip writer
// 尺寸和类型
if !this.compressionConfig.MatchResponse(this.Header().Get("Content-Type"), size, filepath.Ext(this.req.requestPath()), this.req.Format) {
return
}
// 判断Accept是否支持压缩
compressionType, compressionEncoding, ok := this.compressionConfig.MatchAcceptEncoding(this.req.RawReq.Header.Get("Accept-Encoding"))
if !ok {
return
}
this.compressionType = compressionType
// compression writer
var err error = nil
this.gzipWriter, err = gzip.NewWriterLevel(this.writer, int(this.gzipConfig.Level))
this.compressionWriter, err = compressions.NewWriter(this.writer, compressionType, int(this.compressionConfig.Level))
if err != nil {
remotelogs.Error("HTTP_WRITER", err.Error())
return
@@ -304,16 +413,15 @@ func (this *HTTPWriter) prepareGzip(size int64) {
// body copy
if this.bodyCopying {
this.gzipBodyBuffer = bytes.NewBuffer([]byte{})
this.gzipBodyWriter, err = gzip.NewWriterLevel(this.gzipBodyBuffer, int(this.gzipConfig.Level))
this.compressionBodyBuffer = bytes.NewBuffer([]byte{})
this.compressionBodyWriter, err = compressions.NewWriter(this.compressionBodyBuffer, compressionType, int(this.compressionConfig.Level))
if err != nil {
remotelogs.Error("HTTP_WRITER", err.Error())
}
}
header := this.writer.Header()
header.Set("Content-Encoding", "gzip")
header.Set("Transfer-Encoding", "chunked")
header.Set("Content-Encoding", compressionEncoding)
header.Set("Vary", "Accept-Encoding")
header.Del("Content-Length")
}
@@ -329,7 +437,7 @@ func (this *HTTPWriter) prepareCache(size int64) {
return
}
cachePolicy := sharedNodeConfig.HTTPCachePolicy
cachePolicy := this.req.Server.HTTPCachePolicy
if cachePolicy == nil || !cachePolicy.IsOn {
return
}
@@ -344,7 +452,7 @@ func (this *HTTPWriter) prepareCache(size int64) {
return
}
if size >= 0 && ((cacheRef.MaxSizeBytes() > 0 && size > cacheRef.MaxSizeBytes()) ||
(cachePolicy.MaxSizeBytes() > 0 && size > cachePolicy.MaxSizeBytes())) {
(cachePolicy.MaxSizeBytes() > 0 && size > cachePolicy.MaxSizeBytes()) || (cacheRef.MinSizeBytes() > size)) {
return
}
@@ -387,7 +495,11 @@ func (this *HTTPWriter) prepareCache(size int64) {
life = 60
}
expiredAt := utils.UnixTime() + life
cacheWriter, err := storage.OpenWriter(this.req.cacheKey, expiredAt, this.StatusCode())
var cacheKey = this.req.cacheKey
if this.webpIsEncoding {
cacheKey += webpSuffix
}
cacheWriter, err := storage.OpenWriter(cacheKey, expiredAt, this.StatusCode())
if err != nil {
if !caches.CanIgnoreErr(err) {
remotelogs.Error("HTTP_WRITER", "write cache failed: "+err.Error())
@@ -395,9 +507,6 @@ func (this *HTTPWriter) prepareCache(size int64) {
return
}
this.cacheWriter = cacheWriter
if this.gzipWriter != nil {
this.cacheWriter = caches.NewGzipWriter(this.cacheWriter, this.req.cacheKey, expiredAt)
}
// 写入Header
for k, v := range this.Header() {

View File

@@ -0,0 +1,63 @@
// Copyright 2021 Liuxiangchao iwind.liu@gmail.com. All rights reserved.
package nodes
import (
"bufio"
"net"
"net/http"
)
// EmptyResponseWriter 空的响应Writer
type EmptyResponseWriter struct {
header http.Header
parentWriter http.ResponseWriter
statusCode int
}
func NewEmptyResponseWriter(parentWriter http.ResponseWriter) *EmptyResponseWriter {
return &EmptyResponseWriter{
header: http.Header{},
parentWriter: parentWriter,
}
}
func (this *EmptyResponseWriter) Header() http.Header {
return this.header
}
func (this *EmptyResponseWriter) Write(data []byte) (int, error) {
if this.statusCode > 300 && this.parentWriter != nil {
return this.parentWriter.Write(data)
}
return 0, nil
}
func (this *EmptyResponseWriter) WriteHeader(statusCode int) {
this.statusCode = statusCode
if this.statusCode > 300 && this.parentWriter != nil {
var parentHeader = this.parentWriter.Header()
for k, v := range this.header {
parentHeader[k] = v
}
this.parentWriter.WriteHeader(this.statusCode)
}
}
func (this *EmptyResponseWriter) StatusCode() int {
return this.statusCode
}
// Hijack Hijack
func (this *EmptyResponseWriter) Hijack() (conn net.Conn, buf *bufio.ReadWriter, err error) {
if this.parentWriter == nil {
return
}
hijack, ok := this.parentWriter.(http.Hijacker)
if ok {
return hijack.Hijack()
}
return
}

View File

@@ -11,7 +11,7 @@ import (
)
type Listener struct {
group *serverconfigs.ServerGroup
group *serverconfigs.ServerAddressGroup
isListening bool
listener ListenerInterface // 监听器
@@ -22,7 +22,7 @@ func NewListener() *Listener {
return &Listener{}
}
func (this *Listener) Reload(group *serverconfigs.ServerGroup) {
func (this *Listener) Reload(group *serverconfigs.ServerAddressGroup) {
this.locker.Lock()
this.group = group
if this.listener != nil {

View File

@@ -7,7 +7,7 @@ import (
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs/sslconfigs"
"github.com/iwind/TeaGo/lists"
"github.com/iwind/TeaGo/types"
http2 "golang.org/x/net/http2"
"golang.org/x/net/http2"
"sync"
)
@@ -16,7 +16,7 @@ type BaseListener struct {
namedServersLocker sync.RWMutex
namedServers map[string]*NamedServer // 域名 => server
Group *serverconfigs.ServerGroup
Group *serverconfigs.ServerAddressGroup
countActiveConnections int64 // 当前活跃的连接数
}

View File

@@ -1,6 +1,7 @@
package nodes
import (
"context"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"golang.org/x/net/http2"
@@ -9,11 +10,20 @@ import (
"net"
"net/http"
"strings"
"sync"
"sync/atomic"
"time"
)
var httpErrorLogger = log.New(io.Discard, "", 0)
var metricNewConnMap = map[string]bool{} // remoteAddr => bool
var metricNewConnMapLocker = &sync.Mutex{}
type contextKey struct {
key string
}
var HTTPConnContextKey = &contextKey{key: "http-conn"}
type HTTPListener struct {
BaseListener
@@ -39,17 +49,32 @@ func (this *HTTPListener) Serve() error {
this.httpServer = &http.Server{
Addr: this.addr,
Handler: handler,
ReadHeaderTimeout: 3 * time.Second, // TODO 改成可以配置
ReadHeaderTimeout: 2 * time.Second, // TODO 改成可以配置
IdleTimeout: 2 * time.Minute, // TODO 改成可以配置
ErrorLog: httpErrorLogger,
ConnState: func(conn net.Conn, state http.ConnState) {
switch state {
case http.StateNew:
atomic.AddInt64(&this.countActiveConnections, 1)
// 为指标存储连接信息
if sharedNodeConfig.HasHTTPConnectionMetrics() {
metricNewConnMapLocker.Lock()
metricNewConnMap[conn.RemoteAddr().String()] = true
metricNewConnMapLocker.Unlock()
}
case http.StateClosed:
atomic.AddInt64(&this.countActiveConnections, -1)
// 移除指标存储连接信息
metricNewConnMapLocker.Lock()
delete(metricNewConnMap, conn.RemoteAddr().String())
metricNewConnMapLocker.Unlock()
}
},
ConnContext: func(ctx context.Context, c net.Conn) context.Context {
return context.WithValue(ctx, HTTPConnContextKey, c)
},
}
this.httpServer.SetKeepAlivesEnabled(true)
@@ -87,7 +112,7 @@ func (this *HTTPListener) Close() error {
return this.Listener.Close()
}
func (this *HTTPListener) Reload(group *serverconfigs.ServerGroup) {
func (this *HTTPListener) Reload(group *serverconfigs.ServerAddressGroup) {
this.Group = group
this.Reset()

View File

@@ -14,7 +14,7 @@ type ListenerInterface interface {
Close() error
// Reload 重载配置
Reload(serverGroup *serverconfigs.ServerGroup)
Reload(serverGroup *serverconfigs.ServerAddressGroup)
// CountActiveListeners 获取当前活跃的连接数
CountActiveListeners() int

View File

@@ -70,7 +70,7 @@ func (this *ListenerManager) Start(node *nodeconfigs.NodeConfig) error {
groupAddrs := []string{}
availableServerGroups := node.AvailableGroups()
if !node.IsOn {
availableServerGroups = []*serverconfigs.ServerGroup{}
availableServerGroups = []*serverconfigs.ServerAddressGroup{}
}
if len(availableServerGroups) == 0 {

View File

@@ -42,7 +42,7 @@ func (this *TCPListener) Serve() error {
return nil
}
func (this *TCPListener) Reload(group *serverconfigs.ServerGroup) {
func (this *TCPListener) Reload(group *serverconfigs.ServerAddressGroup) {
this.Group = group
this.Reset()
}
@@ -55,6 +55,24 @@ func (this *TCPListener) handleConn(conn net.Conn) error {
if firstServer.ReverseProxy == nil {
return errors.New("no ReverseProxy configured for the server")
}
// 记录域名排行
tlsConn, ok := conn.(*tls.Conn)
var recordStat = false
if ok {
var serverName = tlsConn.ConnectionState().ServerName
if len(serverName) > 0 {
// 统计
stats.SharedTrafficStatManager.Add(firstServer.Id, serverName, 0, 0, 1, 0, 0, 0)
recordStat = true
}
}
// 统计
if !recordStat {
stats.SharedTrafficStatManager.Add(firstServer.Id, "", 0, 0, 1, 0, 0, 0)
}
originConn, err := this.connectOrigin(firstServer.ReverseProxy, conn.RemoteAddr().String())
if err != nil {
return err
@@ -80,7 +98,7 @@ func (this *TCPListener) handleConn(conn net.Conn) error {
}
// 记录流量
stats.SharedTrafficStatManager.Add(firstServer.Id, int64(n), 0, 0, 0)
stats.SharedTrafficStatManager.Add(firstServer.Id, "", int64(n), 0, 0, 0, 0, 0)
}
if err != nil {
closer()

View File

@@ -84,7 +84,7 @@ func (this *UDPListener) Close() error {
return this.Listener.Close()
}
func (this *UDPListener) Reload(group *serverconfigs.ServerGroup) {
func (this *UDPListener) Reload(group *serverconfigs.ServerAddressGroup) {
this.Group = group
this.Reset()
}
@@ -147,6 +147,10 @@ func NewUDPConn(serverId int64, addr net.Addr, proxyConn *net.UDPConn, serverCon
activatedAt: time.Now().Unix(),
isOk: true,
}
// 统计
stats.SharedTrafficStatManager.Add(serverId, "", 0, 0, 1, 0, 0, 0)
go func() {
buffer := bytePool32k.Get()
defer func() {
@@ -164,7 +168,7 @@ func NewUDPConn(serverId int64, addr net.Addr, proxyConn *net.UDPConn, serverCon
}
// 记录流量
stats.SharedTrafficStatManager.Add(serverId, int64(n), 0, 0, 0)
stats.SharedTrafficStatManager.Add(serverId, "", int64(n), 0, 0, 0, 0, 0)
}
if err != nil {
conn.isOk = false

View File

@@ -22,7 +22,7 @@ func (this *UnixListener) Close() error {
return nil
}
func (this *UnixListener) Reload(group *serverconfigs.ServerGroup) {
func (this *UnixListener) Reload(group *serverconfigs.ServerAddressGroup) {
this.Group = group
this.Reset()
}

View File

@@ -6,12 +6,12 @@ import (
"github.com/TeaOSLab/EdgeCommon/pkg/nodeconfigs"
"github.com/TeaOSLab/EdgeCommon/pkg/rpc/pb"
"github.com/TeaOSLab/EdgeCommon/pkg/serverconfigs"
"github.com/TeaOSLab/EdgeNode/internal/apps"
"github.com/TeaOSLab/EdgeNode/internal/caches"
"github.com/TeaOSLab/EdgeNode/internal/configs"
teaconst "github.com/TeaOSLab/EdgeNode/internal/const"
"github.com/TeaOSLab/EdgeNode/internal/events"
"github.com/TeaOSLab/EdgeNode/internal/iplibrary"
"github.com/TeaOSLab/EdgeNode/internal/metrics"
"github.com/TeaOSLab/EdgeNode/internal/remotelogs"
"github.com/TeaOSLab/EdgeNode/internal/rpc"
"github.com/TeaOSLab/EdgeNode/internal/stats"
@@ -20,29 +20,35 @@ import (
"github.com/iwind/TeaGo/Tea"
"github.com/iwind/TeaGo/lists"
"github.com/iwind/TeaGo/logs"
"github.com/iwind/TeaGo/maps"
"github.com/iwind/gosock/pkg/gosock"
"io/ioutil"
"log"
"net"
"os"
"os/exec"
"os/signal"
"runtime"
"syscall"
"sync"
"time"
)
var sharedNodeConfig *nodeconfigs.NodeConfig
var nodeTaskNotify = make(chan bool, 8)
var nodeConfigChangedNotify = make(chan bool, 8)
var nodeConfigUpdatedAt int64
var DaemonIsOn = false
var DaemonPid = 0
// Node 节点
type Node struct {
isLoaded bool
sock *gosock.Sock
locker sync.Mutex
}
func NewNode() *Node {
return &Node{}
return &Node{
sock: gosock.NewTmpSock(teaconst.ProcessName),
}
}
// Test 检查配置
@@ -72,9 +78,6 @@ func (this *Node) Start() {
// 启动事件
events.Notify(events.EventStart)
// 处理信号
this.listenSignals()
// 本地Sock
err := this.listenSock()
if err != nil {
@@ -83,23 +86,30 @@ func (this *Node) Start() {
}
// 读取API配置
tryTimes := 0
for {
err = this.syncConfig()
err = this.syncConfig()
if err != nil {
_, err := nodeconfigs.SharedNodeConfig()
if err != nil {
tryTimes++
// 无本地数据时,会尝试多次读取
tryTimes := 0
for {
err := this.syncConfig()
if err != nil {
tryTimes++
if tryTimes%10 == 0 {
remotelogs.Error("NODE", err.Error())
}
time.Sleep(1 * time.Second)
if tryTimes%10 == 0 {
remotelogs.Error("NODE", err.Error())
}
time.Sleep(1 * time.Second)
// 不做长时间的无意义的重试
if tryTimes > 1000 {
return
// 不做长时间的无意义的重试
if tryTimes > 1000 {
return
}
} else {
break
}
}
} else {
break
}
}
@@ -144,24 +154,16 @@ func (this *Node) Start() {
return
}
// 写入PID
err = apps.WritePid()
if err != nil {
remotelogs.Error("NODE", "write pid failed: "+err.Error())
return
}
// hold住进程
select {}
}
// Daemon 实现守护进程
func (this *Node) Daemon() {
path := os.TempDir() + "/edge-node.sock"
isDebug := lists.ContainsString(os.Args, "debug")
isDebug = true
for {
conn, err := net.DialTimeout("unix", path, 1*time.Second)
conn, err := this.sock.Dial()
if err != nil {
if isDebug {
log.Println("[DAEMON]starting ...")
@@ -221,32 +223,6 @@ func (this *Node) InstallSystemService() error {
return nil
}
// 处理信号
func (this *Node) listenSignals() {
signals := make(chan os.Signal)
signal.Notify(signals, syscall.SIGQUIT)
go func() {
for s := range signals {
switch s {
case syscall.SIGQUIT:
events.Notify(events.EventQuit)
// 监控连接数如果连接数为0则退出进程
go func() {
for {
countActiveConnections := sharedListenerManager.TotalActiveConnections()
if countActiveConnections <= 0 {
os.Exit(0)
return
}
time.Sleep(1 * time.Second)
}
}()
}
}
}()
}
// 循环
func (this *Node) loop() error {
// 检查api.yaml是否存在
@@ -308,6 +284,9 @@ func (this *Node) loop() error {
// 读取API配置
func (this *Node) syncConfig() error {
this.locker.Lock()
defer this.locker.Unlock()
// 检查api.yaml是否存在
apiConfigFile := Tea.ConfigFile("api.yaml")
_, err := os.Stat(apiConfigFile)
@@ -318,7 +297,7 @@ func (this *Node) syncConfig() error {
if os.IsNotExist(clusterErr) {
return err
}
return clusterErr
return errors.New("check cluster config failed: " + clusterErr.Error())
}
} else {
return err
@@ -343,6 +322,7 @@ func (this *Node) syncConfig() error {
if !configResp.IsChanged {
return nil
}
nodeConfigUpdatedAt = time.Now().Unix()
configJSON := configResp.NodeJSON
nodeConfig := &nodeconfigs.NodeConfig{}
@@ -379,8 +359,8 @@ func (this *Node) syncConfig() error {
nodeconfigs.ResetNodeConfig(nodeConfig)
caches.SharedManager.MaxDiskCapacity = nodeConfig.MaxCacheDiskCapacity
caches.SharedManager.MaxMemoryCapacity = nodeConfig.MaxCacheMemoryCapacity
if nodeConfig.HTTPCachePolicy != nil {
caches.SharedManager.UpdatePolicies([]*serverconfigs.HTTPCachePolicy{nodeConfig.HTTPCachePolicy})
if len(nodeConfig.HTTPCachePolicies) > 0 {
caches.SharedManager.UpdatePolicies(nodeConfig.HTTPCachePolicies)
} else {
caches.SharedManager.UpdatePolicies([]*serverconfigs.HTTPCachePolicy{})
}
@@ -389,6 +369,8 @@ func (this *Node) syncConfig() error {
iplibrary.SharedActionManager.UpdateActions(nodeConfig.FirewallActions)
sharedNodeConfig = nodeConfig
metrics.SharedManager.Update(nodeConfig.MetricItems)
// 发送事件
events.Notify(events.EventReload)
@@ -424,6 +406,12 @@ func (this *Node) startSyncTimer() {
remotelogs.Error("NODE", "sync config error: "+err.Error())
continue
}
case <-nodeConfigChangedNotify:
err := this.syncConfig()
if err != nil {
remotelogs.Error("NODE", "sync config error: "+err.Error())
continue
}
}
}
}()
@@ -451,7 +439,7 @@ func (this *Node) checkClusterConfig() error {
return err
}
logs.Println("[NODE]registering node ...")
logs.Println("[NODE]registering node to cluster ...")
resp, err := rpcClient.NodeRPC().RegisterClusterNode(rpcClient.ClusterContext(config.ClusterId, config.Secret), &pb.RegisterClusterNodeRequest{Name: HOSTNAME})
if err != nil {
return err
@@ -483,37 +471,63 @@ func (this *Node) checkClusterConfig() error {
// 监听本地sock
func (this *Node) listenSock() error {
path := os.TempDir() + "/edge-node.sock"
// 检查是否已经存在
_, err := os.Stat(path)
if err == nil {
conn, err := net.Dial("unix", path)
if err != nil {
_ = os.Remove(path)
// 检查是否在运行
if this.sock.IsListening() {
reply, err := this.sock.Send(&gosock.Command{Code: "pid"})
if err == nil {
return errors.New("error: the process is already running, pid: " + maps.NewMap(reply.Params).GetString("pid"))
} else {
_ = conn.Close()
return errors.New("error: the process is already running")
}
}
// 新的监听任务
listener, err := net.Listen("unix", path)
if err != nil {
return err
}
events.On(events.EventQuit, func() {
remotelogs.Println("NODE", "quit unix sock")
_ = listener.Close()
})
// 启动监听
go func() {
for {
_, err := listener.Accept()
if err != nil {
return
this.sock.OnCommand(func(cmd *gosock.Command) {
switch cmd.Code {
case "pid":
_ = cmd.Reply(&gosock.Command{
Code: "pid",
Params: map[string]interface{}{
"pid": os.Getpid(),
},
})
case "stop":
_ = cmd.ReplyOk()
// 退出主进程
events.Notify(events.EventQuit)
os.Exit(0)
case "quit":
_ = cmd.ReplyOk()
_ = this.sock.Close()
events.Notify(events.EventQuit)
// 监控连接数如果连接数为0则退出进程
go func() {
for {
countActiveConnections := sharedListenerManager.TotalActiveConnections()
if countActiveConnections <= 0 {
os.Exit(0)
return
}
time.Sleep(1 * time.Second)
}
}()
}
})
err := this.sock.Listen()
if err != nil {
logs.Println("NODE", err.Error())
}
}()
events.On(events.EventQuit, func() {
logs.Println("NODE", "quit unix sock")
_ = this.sock.Close()
})
return nil
}

View File

@@ -15,6 +15,7 @@ import (
"github.com/iwind/TeaGo/maps"
"github.com/shirou/gopsutil/cpu"
"github.com/shirou/gopsutil/disk"
"golang.org/x/sys/unix"
"os"
"runtime"
"strings"
@@ -67,6 +68,8 @@ func (this *NodeStatusExecutor) update() {
status.ConnectionCount = sharedListenerManager.TotalActiveConnections()
status.CacheTotalDiskSize = caches.SharedManager.TotalDiskSize()
status.CacheTotalMemorySize = caches.SharedManager.TotalMemorySize()
status.TrafficInBytes = inTrafficBytes
status.TrafficOutBytes = outTrafficBytes
// 记录监控数据
monitor.SharedValueQueue.Add(nodeconfigs.NodeValueItemConnections, maps.Map{
@@ -80,6 +83,7 @@ func (this *NodeStatusExecutor) update() {
this.updateMem(status)
this.updateLoad(status)
this.updateDisk(status)
this.updateCacheSpace(status)
status.UpdatedAt = time.Now().Unix()
// 发送数据
@@ -211,3 +215,25 @@ func (this *NodeStatusExecutor) updateDisk(status *nodeconfigs.NodeStatus) {
"maxUsage": status.DiskMaxUsage,
})
}
// 缓存空间
func (this *NodeStatusExecutor) updateCacheSpace(status *nodeconfigs.NodeStatus) {
var result = []maps.Map{}
cachePaths := caches.SharedManager.FindAllCachePaths()
for _, path := range cachePaths {
var stat unix.Statfs_t
err := unix.Statfs(path, &stat)
if err != nil {
return
}
result = append(result, maps.Map{
"path": path,
"total": stat.Blocks * uint64(stat.Bsize),
"avail": stat.Bavail * uint64(stat.Bsize),
"used": (stat.Blocks - stat.Bavail) * uint64(stat.Bsize),
})
}
monitor.SharedValueQueue.Add(nodeconfigs.NodeValueItemCacheDir, maps.Map{
"dirs": result,
})
}

Some files were not shown because too many files have changed in this diff Show More