LF

* Update nat.go
func publicIPTest echo server

.gitignore

@@ -15,4 +15,9 @@ libs/
 openp2p.app.jks
 openp2p.aar
 openp2p-sources.jar
-build.gradle
+wintun/
+wintun.dll
+.vscode/
+app/.idea/
+*_debug_bin*
+cmd/openp2p

README-ZH.md

@@ -19,9 +19,9 @@
 
 [查看详细](#安全性)
 ### 4. 轻量
-文件大小2MB+，运行内存2MB+；全部在应用层实现，没有虚拟网卡，没有内核程序
+文件大小2MB+，运行内存2MB+；它可以仅跑在应用层，或者配合wintun驱动使用组网功能
 ### 5. 跨平台
-因为轻量，所以很容易支持各个平台。支持主流的操作系统：Windows,Linux,MacOS；和主流的cpu架构：386、amd64、arm、arm64、mipsle、mipsle64、mips、mips64
+因为轻量，所以很容易支持各个平台。支持主流的操作系统：Windows,Linux,MacOS；和主流的cpu架构：386、amd64、arm、arm64、mipsle、mipsle64、mips、mips64、s390x、ppc64le
 ### 6. 高效
 P2P直连可以让你的设备跑满带宽。不论你的设备在任何网络环境，无论NAT1-4（Cone或Symmetric），UDP或TCP打洞,UPNP,IPv6都支持。依靠Quic协议优秀的拥塞算法，能在糟糕的网络环境获得高带宽低延时。
 
@@ -101,6 +101,22 @@ cd到代码根目录，执行
 ```
 make
 ```
+手动编译特定系统和架构 
+All GOOS values:
+```
+"aix", "android", "darwin", "dragonfly", "freebsd", "hurd", "illumos", "ios", "js", "linux", "nacl", "netbsd", "openbsd", "plan9", "solaris", "windows", "zos"
+```
+All GOARCH values:
+```
+"386", "amd64", "amd64p32", "arm", "arm64", "arm64be", "armbe", "loong64", "mips", "mips64", "mips64le", "mips64p32", "mips64p32le", "mipsle", "ppc", "ppc64", "ppc64le", "riscv", "riscv64", "s390", "s390x", "sparc", "sparc64", "wasm"
+```
+
+比如linux+amd64
+```
+export GOPROXY=https://goproxy.io,direct
+go mod tidy
+CGO_ENABLED=0 env GOOS=linux GOARCH=amd64 go build -o openp2p --ldflags '-s -w ' -gcflags '-l' -p 8 -installsuffix cgo ./cmd
+```
 
 ## RoadMap
 近期计划：
@@ -110,15 +126,17 @@ make
 4. ~~建立网站，用户可以在网站管理所有P2PApp和设备。查看设备在线状态，升级，增删查改重启P2PApp等~~(100%)
 5. 建立公众号，用户可在微信公众号管理所有P2PApp和设备
 6. 客户端提供WebUI
-7. 支持自有服务器，开源服务器程序
+7. ~~支持自有服务器，开源服务器程序~~(100%)
 8. 共享节点调度模型优化，对不同的运营商优化
-9. 方便二次开发，提供API和lib
+9. ~~方便二次开发，提供API和lib~~(100%)
-10. 应用层支持UDP协议，实现很简单，但UDP应用较少暂不急(100%)
+10. ~~应用层支持UDP协议，实现很简单，但UDP应用较少暂不急~~(100%)
-11. 底层通信支持KCP协议，目前仅支持Quic；KCP专门对延时优化，被游戏加速器广泛使用，可以牺牲一定的带宽降低延时
+11. ~~底层通信支持KCP协议，目前仅支持Quic；KCP专门对延时优化，被游戏加速器广泛使用，可以牺牲一定的带宽降低延时~~(100%)
-12. 支持Android系统，让旧手机焕发青春变成移动网关
+12. ~~支持Android系统，让旧手机焕发青春变成移动网关~~(100%)
-13. 支持Windows网上邻居共享文件
+13. ~~支持Windows网上邻居共享文件~~(100%)
-14. 内网直连优化，用处不大，估计就用户测试时用到
+14. ~~内网直连优化~~(100%)
 15. ~~支持UPNP~~(100%)
+16. ~~支持Android~~(100%)
+17. 支持IOS
 
 远期计划：
 1. 利用区块链技术去中心化，让共享设备的用户有收益，从而促进更多用户共享，达到正向闭环。

README.md

@@ -19,10 +19,10 @@ The code is open source, the P2P tunnel uses TLS1.3+AES double encryption, and t
 [details](#Safety)
 
 ### 4. Lightweight
-2MB+ filesize, 2MB+ memory. It runs at appllication layer, no vitrual NIC, no kernel driver.
+2MB+ filesize, 2MB+ memory. It could only runs at application layer, or uses wintun driver for SDWAN.
 
 ### 5. Cross-platform
-Benefit from lightweight, it easily supports most of major OS, like Windows, Linux, MacOS, also most of CPU architecture, like 386、amd64、arm、arm64、mipsle、mipsle64、mips、mips64.
+Benefit from lightweight, it easily supports most of major OS, like Windows, Linux, MacOS, also most of CPU architecture, like 386、amd64、arm、arm64、mipsle、mipsle64、mips、mips64、s390x、ppc64le.
 
 ### 6. Efficient
 P2P direct connection lets your devices make good use of bandwidth.  Your device can be connected in any network environments, even supports NAT1-4 (Cone or Symmetric),UDP or TCP punching,UPNP,IPv6.  Relying on the excellent congestion algorithm of the Quic protocol, high bandwidth and low latency can be obtained in a bad network environment.
@@ -109,6 +109,23 @@ cd root directory of the socure code and execute
 make
 ```
 
+build specified os and arch.  
+All GOOS values:
+```
+"aix", "android", "darwin", "dragonfly", "freebsd", "hurd", "illumos", "ios", "js", "linux", "nacl", "netbsd", "openbsd", "plan9", "solaris", "windows", "zos"
+```
+All GOARCH values:
+```
+"386", "amd64", "amd64p32", "arm", "arm64", "arm64be", "armbe", "loong64", "mips", "mips64", "mips64le", "mips64p32", "mips64p32le", "mipsle", "ppc", "ppc64", "ppc64le", "riscv", "riscv64", "s390", "s390x", "sparc", "sparc64", "wasm"
+```
+
+For example linux+amd64
+```
+export GOPROXY=https://goproxy.io,direct
+go mod tidy
+CGO_ENABLED=0 env GOOS=linux GOARCH=amd64 go build -o openp2p --ldflags '-s -w ' -gcflags '-l' -p 8 -installsuffix cgo ./cmd
+```
+
 ## RoadMap
 Short-Term:
 1. ~~Support IPv6.~~(100%)
@@ -117,16 +134,17 @@ Short-Term:
 4. ~~Build website, users can manage all P2PApp and devices via it. View devices' online status, upgrade, restart or CURD P2PApp .~~(100%)
 5. Provide wechat official account, user can manage P2PApp nodes and deivce as same as website.
 6. Provide WebUI on client side.
-7. Support private server, open source server program.
+7. ~~Support private server, open source server program.~~(100%)
 8. Optimize our share scheduling model for different network operators.
-9. Provide REST APIs and libary for secondary development.
+9. ~~Provide REST APIs and libary for secondary development.~~(100%)
 10. ~~Support UDP at application layer, it is easy to implement but not urgent due to only a few applicaitons using UDP protocol.~~(100%)
-11. Support KCP protocol underlay, currently support Quic only. KCP focus on delay optimization,which has been widely used as game accelerator,it can sacrifice part of bandwidth to reduce timelag. 
+11. ~~Support KCP protocol underlay, currently support Quic only. KCP focus on delay optimization,which has been widely used as game accelerator,it can sacrifice part of bandwidth to reduce timelag. ~~(100%)
-12. Support Android platform, let the phones to be mobile gateway.
+12. ~~Support Android platform, let the phones to be mobile gateway.~~(100%)
-13. Support SMB Windows neighborhood.
+13. ~~Support SMB Windows neighborhood.~~(100%)
-14. Direct connection on intranet, for testing.
+14. ~~Direct connection on intranet, for testing.~~(100%)
 15. ~~Support UPNP.~~(100%)
-
+16. ~~Support Android~~(100%)
+17. Support IOS
 
 Long-Term:
 1. Use blockchain technology to decentralize, so that users who share equipment have benefits, thereby promoting more users to share, and achieving a positive closed loop.

USAGE-ZH.md

@@ -1,99 +1,108 @@
-# 手动运行说明
+# 手动运行说明
-大部分情况通过<https://console.openp2p.cn> 操作即可。有些情况需要手动运行  
+大部分情况通过<https://console.openp2p.cn> 操作即可。有些情况需要手动运行  
-> :warning: 本文所有命令, Windows环境使用"openp2p.exe", Linux环境使用"./openp2p"
+> :warning: 本文所有命令, Windows环境使用"openp2p.exe", Linux环境使用"./openp2p"
-
+
-
+
-## 安装和监听
+## 安装和监听
-```
+```
-./openp2p install -node OFFICEPC1 -token TOKEN  
+./openp2p install -node OFFICEPC1 -token TOKEN  
-或
+或
-./openp2p -d -node OFFICEPC1 -token TOKEN  
+./openp2p -d -node OFFICEPC1 -token TOKEN  
-# 注意Windows系统把“./openp2p” 换成“openp2p.exe”
+# 注意Windows系统把“./openp2p” 换成“openp2p.exe”
-```
+```
->* install: 安装模式【推荐】，会安装成系统服务，这样它就能随系统自动启动
+>* install: 安装模式【推荐】，会安装成系统服务，这样它就能随系统自动启动
->* -d: daemon模式。发现worker进程意外退出就会自动启动新的worker进程
+>* -d: daemon模式。发现worker进程意外退出就会自动启动新的worker进程
->* -node: 独一无二的节点名字，唯一标识
+>* -node: 独一无二的节点名字，唯一标识
->* -token: 在<console.openp2p.cn>“我的”里面找到
+>* -token: 在<console.openp2p.cn>“我的”里面找到
->* -sharebandwidth: 作为共享节点时提供带宽，默认10mbps. 如果是光纤大带宽，设置越大效果越好. 0表示不共享，该节点只在私有的P2P网络使用。不加入共享的P2P网络，这样也意味着无法使用别人的共享节点
+>* -sharebandwidth: 作为共享节点时提供带宽，默认10mbps. 如果是光纤大带宽，设置越大效果越好. 0表示不共享，该节点只在私有的P2P网络使用。不加入共享的P2P网络，这样也意味着无法使用别人的共享节点
->* -loglevel: 需要查看更多调试日志，设置0；默认是1
+>* -loglevel: 需要查看更多调试日志，设置0；默认是1
-
+
-### 在docker容器里运行openp2p
+### 在docker容器里运行openp2p
-我们暂时还没提供官方docker镜像，你可以在随便一个容器里运行
+我们暂时还没提供官方docker镜像，你可以在随便一个容器里运行
-```
+```
-nohup ./openp2p -d -node OFFICEPC1 -token TOKEN  &
+nohup ./openp2p -d -node OFFICEPC1 -token TOKEN  &
-#这里由于一般的镜像都精简过，install系统服务会失败，所以使用直接daemon模式后台运行
+#这里由于一般的镜像都精简过，install系统服务会失败，所以使用直接daemon模式后台运行
-```
+```
-## 连接
+## 连接
-```
+```
-./openp2p -d -node HOMEPC123 -token TOKEN -appname OfficeWindowsRemote -peernode OFFICEPC1 -dstip 127.0.0.1 -dstport 3389 -srcport 23389
+./openp2p -d -node HOMEPC123 -token TOKEN -appname OfficeWindowsRemote -peernode OFFICEPC1 -dstip 127.0.0.1 -dstport 3389 -srcport 23389
-使用配置文件，建立多个P2PApp
+使用配置文件，建立多个P2PApp
-./openp2p -d   
+./openp2p -d   
-```
+```
->* -appname: 这个P2P应用名字
+>* -appname: 这个P2P应用名字
->* -peernode: 目标节点名字
+>* -peernode: 目标节点名字
->* -dstip: 目标服务地址，默认本机127.0.0.1
+>* -dstip: 目标服务地址，默认本机127.0.0.1
->* -dstport: 目标服务端口，常见的如windows远程桌面3389，Linux ssh 22
+>* -dstport: 目标服务端口，常见的如windows远程桌面3389，Linux ssh 22
->* -protocol: 目标服务协议 tcp、udp
+>* -protocol: 目标服务协议 tcp、udp
-
+
-## 配置文件
+## 配置文件
-一般保存在当前目录，安装模式下会保存到 `C:\Program Files\OpenP2P\config.json` 或 `/usr/local/openp2p/config.json`
+一般保存在当前目录，安装模式下会保存到 `C:\Program Files\OpenP2P\config.json` 或 `/usr/local/openp2p/config.json`
-希望修改参数，或者配置多个P2PApp可手动修改配置文件
+希望修改参数，或者配置多个P2PApp可手动修改配置文件
-
+
-配置实例
+配置实例
-```
+```
-{
+{
-  "network": {
+  "network": {
-    "Node": "hhd1207-222",
+    "Node": "YOUR-NODE-NAME",
-    "Token": "TOKEN",
+    "Token": "TOKEN",
-    "ShareBandwidth": 0,
+    "ShareBandwidth": 0,
-    "ServerHost": "api.openp2p.cn",
+    "ServerHost": "api.openp2p.cn",
-    "ServerPort": 27183,
+    "ServerPort": 27183,
-    "UDPPort1": 27182,
+    "UDPPort1": 27182,
-    "UDPPort2": 27183
+    "UDPPort2": 27183
-  },
+  },
-  "apps": [
+  "apps": [
-    {
+    {
-      "AppName": "OfficeWindowsPC",
+      "AppName": "OfficeWindowsPC",
-      "Protocol": "tcp",
+      "Protocol": "tcp",
-      "SrcPort": 23389,
+      "SrcPort": 23389,
-      "PeerNode": "OFFICEPC1",
+      "PeerNode": "OFFICEPC1",
-      "DstPort": 3389,
+      "DstPort": 3389,
-      "DstHost": "localhost",
+      "DstHost": "localhost",
-    },
+    },
-    {
+    {
-      "AppName": "OfficeServerSSH",
+      "AppName": "OfficeServerSSH",
-      "Protocol": "tcp",
+      "Protocol": "tcp",
-      "SrcPort": 22,
+      "SrcPort": 22,
-      "PeerNode": "OFFICEPC1",
+      "PeerNode": "OFFICEPC1",
-      "DstPort": 22,
+      "DstPort": 22,
-      "DstHost": "192.168.1.5",
+      "DstHost": "192.168.1.5",
-    }
+    }
-  ]
+  ]
-}
+}
-```
+```
-
+
-## 升级客户端
+## 升级客户端
-```
+```
-# update local client
+# update local client
-./openp2p update  
+./openp2p update  
-# update remote client
+# update remote client
-curl --insecure 'https://api.openp2p.cn:27183/api/v1/device/YOUR-NODE-NAME/update?user=&password='
+curl --insecure 'https://api.openp2p.cn:27183/api/v1/device/YOUR-NODE-NAME/update?user=&password='
-```
+```
-
+
-Windows系统需要设置防火墙放行本程序，程序会自动设置，如果设置失败会影响连接功能。
+Windows系统需要设置防火墙放行本程序，程序会自动设置，如果设置失败会影响连接功能。
-Linux系统（Ubuntu和CentOS7）的防火墙默认配置均不会有影响，如果不行可尝试关闭防火墙
+Linux系统（Ubuntu和CentOS7）的防火墙默认配置均不会有影响，如果不行可尝试关闭防火墙
-```
+```
-systemctl stop firewalld.service
+systemctl stop firewalld.service
-systemctl start firewalld.service
+systemctl start firewalld.service
-firewall-cmd --state
+firewall-cmd --state
-```
+```
-
+## 停止
-## 卸载
+TODO: windows linux macos
-```
+## 卸载
-./openp2p uninstall
+```
-# 已安装时
+./openp2p uninstall
-# windows
+# 已安装时
-C:\Program Files\OpenP2P\openp2p.exe uninstall
+# windows
-# linux,macos
+C:\Program Files\OpenP2P\openp2p.exe uninstall
-sudo /usr/local/openp2p/openp2p uninstall
+# linux,macos
-```
+sudo /usr/local/openp2p/openp2p uninstall
+```
+
+## Docker运行
+```
+# 把YOUR-TOKEN和YOUR-NODE-NAME替换成自己的
+docker run -d --restart=always --net host --name openp2p-client -e OPENP2P_TOKEN=YOUR-TOKEN -e OPENP2P_NODE=YOUR-NODE-NAME  openp2pcn/openp2p-client:latest 
+OR
+docker run -d --restart=always --net host --name openp2p-client  openp2pcn/openp2p-client:latest -token YOUR-TOKEN -node YOUR-NODE-NAME
+```

USAGE.md

@@ -1,101 +1,109 @@
-
+
-
+
-# Parameters details
+# Parameters details
-In most cases, you can operate it through <https://console.openp2p.cn>. In some cases it is necessary to run manually
+In most cases, you can operate it through <https://console.openp2p.cn>. In some cases it is necessary to run manually
-> :warning: all commands in this doc, Windows env uses "openp2p.exe", Linux env uses "./openp2p" 
+> :warning: all commands in this doc, Windows env uses "openp2p.exe", Linux env uses "./openp2p" 
-
+
-
+
-## Install and Listen
+## Install and Listen
-```
+```
-./openp2p install -node OFFICEPC1 -token TOKEN  
+./openp2p install -node OFFICEPC1 -token TOKEN  
-Or
+Or
-./openp2p -d -node OFFICEPC1 -token TOKEN  
+./openp2p -d -node OFFICEPC1 -token TOKEN  
-
+
-```
+```
->* install: [recommand] will install as system service. So it will autorun when system booting.
+>* install: [recommand] will install as system service. So it will autorun when system booting.
->* -d: daemon mode run once. When the worker process is found to exit unexpectedly, a new worker process will be automatically started
+>* -d: daemon mode run once. When the worker process is found to exit unexpectedly, a new worker process will be automatically started
->* -node: Unique node name, unique identification
+>* -node: Unique node name, unique identification
->* -token: See <console.openp2p.cn> "Profile"
+>* -token: See <console.openp2p.cn> "Profile"
->* -sharebandwidth: Provides bandwidth when used as a shared node, the default is 10mbps. If it is a large bandwidth of optical fiber, the larger the setting, the better the effect. 0 means not shared, the node is only used in a private P2P network. Do not join the shared P2P network, which also means that you CAN NOT use other people’s shared nodes
+>* -sharebandwidth: Provides bandwidth when used as a shared node, the default is 10mbps. If it is a large bandwidth of optical fiber, the larger the setting, the better the effect. 0 means not shared, the node is only used in a private P2P network. Do not join the shared P2P network, which also means that you CAN NOT use other people’s shared nodes
->* -loglevel: Need to view more debug logs, set 0; the default is 1
+>* -loglevel: Need to view more debug logs, set 0; the default is 1
-
+
-### Run in Docker container
+### Run in Docker container
-We don't provide official docker image yet, you can run it in any container
+We don't provide official docker image yet, you can run it in any container
-```
+```
-nohup ./openp2p -d -node OFFICEPC1 -token TOKEN  &
+nohup ./openp2p -d -node OFFICEPC1 -token TOKEN  &
-# Since many docker images have been simplified, the install system service will fail, so the daemon mode is used to run in the background
+# Since many docker images have been simplified, the install system service will fail, so the daemon mode is used to run in the background
-```
+```
-
+
-## Connect
+## Connect
-```
+```
-./openp2p -d -node HOMEPC123 -token TOKEN -appname OfficeWindowsRemote -peernode OFFICEPC1 -dstip 127.0.0.1 -dstport 3389 -srcport 23389
+./openp2p -d -node HOMEPC123 -token TOKEN -appname OfficeWindowsRemote -peernode OFFICEPC1 -dstip 127.0.0.1 -dstport 3389 -srcport 23389
-Create multiple P2PApp by config file
+Create multiple P2PApp by config file
-./openp2p -d    
+./openp2p -d    
-```
+```
->* -appname: This P2PApp name
+>* -appname: This P2PApp name
->* -peernode: Target node name
+>* -peernode: Target node name
->* -dstip: Target service address, default local 127.0.0.1
+>* -dstip: Target service address, default local 127.0.0.1
->* -dstport: Target service port, such as windows remote desktop 3389, Linux ssh 22
+>* -dstport: Target service port, such as windows remote desktop 3389, Linux ssh 22
->* -protocol: Target service protocol tcp, udp
+>* -protocol: Target service protocol tcp, udp
-
+
-## Config file
+## Config file
-Generally saved in the current directory, in installation mode it will be saved to `C:\Program Files\OpenP2P\config.json` or `/usr/local/openp2p/config.json`
+Generally saved in the current directory, in installation mode it will be saved to `C:\Program Files\OpenP2P\config.json` or `/usr/local/openp2p/config.json`
-If you want to modify the parameters, or configure multiple P2PApps, you can manually modify the configuration file
+If you want to modify the parameters, or configure multiple P2PApps, you can manually modify the configuration file
-
+
-Configuration example
+Configuration example
-```
+```
-{
+{
-  "network": {
+  "network": {
-    "Node": "hhd1207-222",
+    "Node": "YOUR-NODE-NAME",
-    "Token": "TOKEN",
+    "Token": "TOKEN",
-    "ShareBandwidth": 0,
+    "ShareBandwidth": 0,
-    "ServerHost": "api.openp2p.cn",
+    "ServerHost": "api.openp2p.cn",
-    "ServerPort": 27183,
+    "ServerPort": 27183,
-    "UDPPort1": 27182,
+    "UDPPort1": 27182,
-    "UDPPort2": 27183
+    "UDPPort2": 27183
-  },
+  },
-  "apps": [
+  "apps": [
-    {
+    {
-      "AppName": "OfficeWindowsPC",
+      "AppName": "OfficeWindowsPC",
-      "Protocol": "tcp",
+      "Protocol": "tcp",
-      "SrcPort": 23389,
+      "SrcPort": 23389,
-      "PeerNode": "OFFICEPC1",
+      "PeerNode": "OFFICEPC1",
-      "DstPort": 3389,
+      "DstPort": 3389,
-      "DstHost": "localhost",
+      "DstHost": "localhost",
-    },
+    },
-    {
+    {
-      "AppName": "OfficeServerSSH",
+      "AppName": "OfficeServerSSH",
-      "Protocol": "tcp",
+      "Protocol": "tcp",
-      "SrcPort": 22,
+      "SrcPort": 22,
-      "PeerNode": "OFFICEPC1",
+      "PeerNode": "OFFICEPC1",
-      "DstPort": 22,
+      "DstPort": 22,
-      "DstHost": "192.168.1.5",
+      "DstHost": "192.168.1.5",
-    }
+    }
-  ]
+  ]
-}
+}
-```
+```
-## Client update
+## Client update
-```
+```
-# update local client
+# update local client
-./openp2p update  
+./openp2p update  
-# update remote client
+# update remote client
-curl --insecure 'https://api.openp2p.cn:27183/api/v1/device/YOUR-NODE-NAME/update?user=&password='
+curl --insecure 'https://api.openp2p.cn:27183/api/v1/device/YOUR-NODE-NAME/update?user=&password='
-```
+```
-
+
-Windows system needs to set up firewall for this program, the program will automatically set the firewall, if the setting fails, the UDP punching will be affected.  
+Windows system needs to set up firewall for this program, the program will automatically set the firewall, if the setting fails, the UDP punching will be affected.  
-The default firewall configuration of Linux system (Ubuntu and CentOS7) will not have any effect, if not, you can try to turn off the firewall
+The default firewall configuration of Linux system (Ubuntu and CentOS7) will not have any effect, if not, you can try to turn off the firewall
-```
+```
-systemctl stop firewalld.service
+systemctl stop firewalld.service
-systemctl start firewalld.service
+systemctl start firewalld.service
-firewall-cmd --state
+firewall-cmd --state
-```
+```
-
+
-## Uninstall
+## Uninstall
-```
+```
-./openp2p uninstall
+./openp2p uninstall
-# when already installed
+# when already installed
-# windows
+# windows
-C:\Program Files\OpenP2P\openp2p.exe uninstall
+C:\Program Files\OpenP2P\openp2p.exe uninstall
-# linux,macos
+# linux,macos
-sudo /usr/local/openp2p/openp2p uninstall
+sudo /usr/local/openp2p/openp2p uninstall
+```
+
+## Run with Docker
+```
+# Replace YOUR-TOKEN and YOUR-NODE-NAME with yours
+docker run -d --net host --name openp2p-client -e OPENP2P_TOKEN=YOUR-TOKEN -e OPENP2P_NODE=YOUR-NODE-NAME  openp2pcn/openp2p-client:latest 
+OR
+docker run -d --net host --name openp2p-client  openp2pcn/openp2p-client:latest -token YOUR-TOKEN -node YOUR-NODE-NAME
 ```

app/.idea/runConfigurations.xml

@@ -1,10 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<project version="4">
-  <component name="RunConfigurationProducerService">
-    <option name="ignoredProducers">
-      <set>
-        <option value="com.android.tools.idea.compose.preview.runconfiguration.ComposePreviewRunConfigurationProducer" />
-      </set>
-    </option>
-  </component>
-</project>

app/README.md

@@ -0,0 +1,32 @@
+## Build
+depends on openjdk 11, gradle 8.1.3, ndk 21
+```
+
+go install golang.org/x/mobile/cmd/gomobile@latest
+gomobile init
+go get -v golang.org/x/mobile/bind
+cd core
+gomobile bind -target android -v
+if [[ $? -ne 0 ]]; then
+    echo "build error"
+    exit 9
+fi
+echo "build ok"
+cp openp2p.aar openp2p-sources.jar ../app/app/libs
+echo "copy to APP libs"
+
+edit app/app/build.gradle 
+```
+signingConfigs {
+        release {
+            storeFile file('YOUR-JKS-PATH')
+            storePassword 'YOUR-PASSWORD'
+            keyAlias 'openp2p.keys'
+            keyPassword 'YOUR-PASSWORD'
+        }
+    }
+```
+cd ../app
+./gradlew build
+
+```

app/app/build.gradle

@@ -0,0 +1,63 @@
+plugins {
+    id 'com.android.application'
+    id 'kotlin-android'
+}
+
+android {
+    signingConfigs {
+        release {
+            storeFile file('C:\\work\\src\\openp2p-client\\app\\openp2p.jks')
+            storePassword 'YOUR-PASSWORD'
+            keyAlias 'openp2p.keys'
+            keyPassword 'YOUR-PASSWORD'
+        }
+    }
+    compileSdkVersion 31
+    buildToolsVersion "30.0.3"
+
+    defaultConfig {
+        applicationId "cn.openp2p"
+        minSdkVersion 16
+        targetSdkVersion 31
+        versionCode 1
+        versionName "2718281828"
+
+        testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
+        namespace "cn.openp2p"
+    }
+
+    buildTypes {
+        release {
+            minifyEnabled true
+            shrinkResources true
+            proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
+            signingConfig signingConfigs.release
+        }
+    }
+    compileOptions {
+        sourceCompatibility JavaVersion.VERSION_1_8
+        targetCompatibility JavaVersion.VERSION_1_8
+    }
+    kotlinOptions {
+        jvmTarget = '1.8'
+    }
+    buildFeatures {
+        viewBinding true
+    }
+}
+
+dependencies {
+    implementation fileTree(dir: "libs", include: ["*.jar", "*.aar"])
+    implementation "org.jetbrains.kotlin:kotlin-stdlib:$kotlin_version"
+    implementation 'androidx.core:core-ktx:1.3.1'
+    implementation 'androidx.appcompat:appcompat:1.2.0'
+    implementation 'com.google.android.material:material:1.2.1'
+    implementation 'androidx.annotation:annotation:1.1.0'
+    implementation 'androidx.constraintlayout:constraintlayout:2.0.1'
+    implementation 'androidx.lifecycle:lifecycle-livedata-ktx:2.2.0'
+    implementation 'androidx.lifecycle:lifecycle-viewmodel-ktx:2.2.0'
+    testImplementation 'junit:junit:4.+'
+    androidTestImplementation 'androidx.test.ext:junit:1.1.2'
+    androidTestImplementation 'androidx.test.espresso:espresso-core:3.3.0'
+    implementation files('libs\\openp2p-sources.jar')
+}

app/app/src/main/AndroidManifest.xml

@@ -1,12 +1,11 @@
 <?xml version="1.0" encoding="utf-8"?>
-<manifest xmlns:android="http://schemas.android.com/apk/res/android"
+<manifest xmlns:android="http://schemas.android.com/apk/res/android">
-    package="cn.openp2p">
 
     <uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
     <uses-permission android:name="android.permission.INTERNET" />
     <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
     <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
-
+    <uses-permission android:name="android.permission.RECEIVE_BOOT_COMPLETED" />
 
     <application
         android:allowBackup="true"
@@ -27,13 +26,21 @@
 
         <activity
             android:name=".ui.login.LoginActivity"
-            android:label="@string/app_name">
+            android:label="@string/app_name"
+            android:exported="true">
             <intent-filter>
                 <action android:name="android.intent.action.MAIN" />
 
                 <category android:name="android.intent.category.LAUNCHER" />
             </intent-filter>
         </activity>
+        <receiver android:name="BootReceiver"
+            android:exported="true"
+            android:enabled="true">
+            <intent-filter>
+                <action android:name="android.intent.action.BOOT_COMPLETED"/>
+            </intent-filter>
+        </receiver>
     </application>
 
 </manifest>

app/app/src/main/java/cn/openp2p/BootReceiver.kt

@@ -0,0 +1,27 @@
+package cn.openp2p
+
+import android.content.BroadcastReceiver
+import android.content.Context
+import android.content.Intent
+import android.net.VpnService
+import android.os.Build
+import android.os.Bundle
+import android.util.Log
+import cn.openp2p.ui.login.LoginActivity
+class BootReceiver : BroadcastReceiver() {
+    override fun onReceive(context: Context, intent: Intent) {
+//        Logger.log("pp onReceive "+intent.action.toString())
+        Log.i("onReceive","start "+intent.action.toString())
+        // if (Intent.ACTION_BOOT_COMPLETED == intent.action) {
+        //     Log.i("onReceive","match "+intent.action.toString())
+        //     VpnService.prepare(context)
+        //     val intent = Intent(context, OpenP2PService::class.java)
+        //     if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
+        //         context.startForegroundService(intent)
+        //     } else {
+        //         context.startService(intent)
+        //     }
+        // }
+        Log.i("onReceive","end "+intent.action.toString())
+    }
+}

app/app/src/main/java/cn/openp2p/OpenP2PService.kt

@@ -4,9 +4,12 @@ import android.app.*
 import android.content.Context
 import android.content.Intent
 import android.graphics.Color
+import java.io.IOException
+import android.net.VpnService
 import android.os.Binder
 import android.os.Build
 import android.os.IBinder
+import android.os.ParcelFileDescriptor
 import android.util.Log
 import androidx.annotation.RequiresApi
 import androidx.core.app.NotificationCompat
@@ -14,9 +17,22 @@ import cn.openp2p.ui.login.LoginActivity
 import kotlinx.coroutines.GlobalScope
 import kotlinx.coroutines.launch
 import openp2p.Openp2p
+import java.io.FileInputStream
+import java.io.FileOutputStream
+import java.nio.ByteBuffer
+import kotlinx.coroutines.*
+import org.json.JSONObject
+
+data class Node(val name: String, val ip: String, val resource: String? = null)
 
 
-class OpenP2PService : Service() {
+data class Network(
+    val id: Long,
+    val name: String,
+    val gateway: String,
+    val Nodes: List<Node>
+)
+class OpenP2PService : VpnService() {
     companion object {
         private val LOG_TAG = OpenP2PService::class.simpleName
     }
@@ -29,10 +45,12 @@ class OpenP2PService : Service() {
     private lateinit var network: openp2p.P2PNetwork
     private lateinit var mToken: String
     private var running:Boolean =true
+    private var sdwanRunning:Boolean =false
+    private var vpnInterface: ParcelFileDescriptor? = null
+    private  var sdwanJob: Job? = null
     override fun onCreate() {
         Log.i(LOG_TAG, "onCreate - Thread ID = " + Thread.currentThread().id)
-        var channelId: String? = null
+        var channelId = if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
-        channelId = if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
             createNotificationChannel("kim.hsl", "ForegroundService")
         } else {
             ""
@@ -41,7 +59,7 @@ class OpenP2PService : Service() {
 
         val pendingIntent = PendingIntent.getActivity(
             this, 0,
-            notificationIntent, 0
+            notificationIntent, PendingIntent.FLAG_IMMUTABLE
         )
 
         val notification = channelId?.let {
@@ -54,7 +72,7 @@ class OpenP2PService : Service() {
 
         startForeground(1337, notification)
         super.onCreate()
-
+        refreshSDWAN()
     }
 
     override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
@@ -62,23 +80,172 @@ class OpenP2PService : Service() {
             LOG_TAG,
             "onStartCommand - startId = " + startId + ", Thread ID = " + Thread.currentThread().id
         )
+        startOpenP2P(null)
         return super.onStartCommand(intent, flags, startId)
     }
 
     override fun onBind(p0: Intent?): IBinder? {
-
         val token = p0?.getStringExtra("token")
-        Log.i(LOG_TAG, "onBind - Thread ID = " + Thread.currentThread().id + token)
+        Log.i(LOG_TAG, "onBind token=$token")
+        startOpenP2P(token)
+        return binder
+    }
+
+    private fun startOpenP2P(token : String?): Boolean {
+        if (sdwanRunning) {
+            return true
+        }
+
+        Log.i(LOG_TAG, "startOpenP2P - Thread ID = " + Thread.currentThread().id + token)
+        val oldToken = Openp2p.getToken(getExternalFilesDir(null).toString())
+        Log.i(LOG_TAG, "startOpenP2P oldtoken=$oldToken newtoken=$token")
+        if (oldToken=="0" && token==null){
+            return false
+        }
+        sdwanRunning = true
+        //        runSDWAN()
         GlobalScope.launch {
-            network = Openp2p.runAsModule(getExternalFilesDir(null).toString(), token, 0, 1)
+            network = Openp2p.runAsModule(
+                getExternalFilesDir(null).toString(),
+                token,
+                0,
+                1
+            ) // /storage/emulated/0/Android/data/cn.openp2p/files/
             val isConnect = network.connect(30000)  // ms
-            Log.i(OpenP2PService.LOG_TAG, "login result: " + isConnect.toString());
+            Log.i(LOG_TAG, "login result: " + isConnect.toString());
             do {
                 Thread.sleep(1000)
-            }while(network.connect(30000)&&running)
+            } while (network.connect(30000) && running)
             stopSelf()
         }
-        return binder
+        return false
+    }
+
+    private fun refreshSDWAN() {
+        GlobalScope.launch {
+            Log.i(OpenP2PService.LOG_TAG, "refreshSDWAN start");
+            while (true) {
+                Log.i(OpenP2PService.LOG_TAG, "waiting new sdwan config");
+                val buf = ByteArray(4096)
+                val buffLen = Openp2p.getAndroidSDWANConfig(buf)
+                Log.i(OpenP2PService.LOG_TAG, "closing running sdwan instance");
+                sdwanRunning = false
+                vpnInterface?.close()
+                vpnInterface = null
+                Thread.sleep(10000)
+                runSDWAN(buf.copyOfRange(0,buffLen.toInt() ))
+            }
+            Log.i(OpenP2PService.LOG_TAG, "refreshSDWAN end");
+        }
+    }
+    private suspend fun readTunLoop() {
+        val inputStream = FileInputStream(vpnInterface?.fileDescriptor).channel
+        if (inputStream==null){
+            Log.i(OpenP2PService.LOG_TAG, "open FileInputStream error: ");
+            return
+        }
+        Log.d(LOG_TAG, "read tun loop start")
+        val buffer = ByteBuffer.allocate(4096)
+        val byteArrayRead = ByteArray(4096)
+        while (sdwanRunning) {
+            buffer.clear()
+            val readBytes = inputStream.read(buffer)
+            if (readBytes <= 0) {
+//                Log.i(OpenP2PService.LOG_TAG, "inputStream.read error: ")
+                delay(1)
+                continue
+            }
+            buffer.flip()
+            buffer.get(byteArrayRead,0,readBytes)
+            Log.i(OpenP2PService.LOG_TAG, String.format("Openp2p.androidRead: %d", readBytes))
+            Openp2p.androidRead(byteArrayRead, readBytes.toLong())
+
+        }
+        Log.d(LOG_TAG, "read tun loop end")
+    }
+
+    private fun runSDWAN(buf:ByteArray) {
+        sdwanRunning=true
+        sdwanJob=GlobalScope.launch(context = Dispatchers.IO) {
+            Log.i(OpenP2PService.LOG_TAG, "runSDWAN start:${buf.decodeToString()}");
+            try{
+                var builder = Builder()
+                val jsonObject = JSONObject(buf.decodeToString())
+                val id = jsonObject.getLong("id")
+                val name = jsonObject.getString("name")
+                val gateway = jsonObject.getString("gateway")
+                val nodesArray = jsonObject.getJSONArray("Nodes")
+
+                val nodesList = mutableListOf<JSONObject>()
+                for (i in 0 until nodesArray.length()) {
+                    nodesList.add(nodesArray.getJSONObject(i))
+                }
+
+                val myNodeName = Openp2p.getAndroidNodeName()
+                Log.i(OpenP2PService.LOG_TAG, "getAndroidNodeName:${myNodeName}");
+                val nodeList = nodesList.map {
+                    val nodeName = it.getString("name")
+                    val nodeIp = it.getString("ip")
+                    if (nodeName==myNodeName){
+                        builder.addAddress(nodeIp, 24)
+                    }
+                    val nodeResource = if (it.has("resource")) it.getString("resource") else null
+                    val parts = nodeResource?.split("/")
+                    if (parts?.size == 2) {
+                        val ipAddress = parts[0]
+                        val subnetMask = parts[1]
+                        builder.addRoute(ipAddress, subnetMask.toInt())
+                        Log.i(OpenP2PService.LOG_TAG, "sdwan addRoute:${ipAddress},${subnetMask.toInt()}");
+                    }
+                    Node(nodeName, nodeIp, nodeResource)
+                }
+
+                val network = Network(id, name, gateway, nodeList)
+                println(network)
+                Log.i(OpenP2PService.LOG_TAG, "onBind");
+                builder.addDnsServer("8.8.8.8")
+                builder.addRoute("10.2.3.0", 24)
+//            builder.addRoute("0.0.0.0", 0);
+                builder.setSession(LOG_TAG!!)
+                builder.setMtu(1420)
+                vpnInterface = builder.establish()
+                if (vpnInterface==null){
+                    Log.e(OpenP2PService.LOG_TAG, "start vpnservice error: ");
+                }
+               val outputStream = FileOutputStream(vpnInterface?.fileDescriptor).channel
+               if (outputStream==null){
+                   Log.e(OpenP2PService.LOG_TAG, "open FileOutputStream error: ");
+                   return@launch
+               }
+
+               val byteArrayWrite = ByteArray(4096)
+                launch {
+                        readTunLoop()
+                    }
+
+                Log.d(LOG_TAG, "write tun loop start")
+               while (sdwanRunning) {
+                   val len = Openp2p.androidWrite(byteArrayWrite)
+                   Log.i(OpenP2PService.LOG_TAG, String.format("Openp2p.androidWrite: %d",len));
+                   val writeBytes = outputStream?.write(ByteBuffer.wrap(byteArrayWrite))
+                   if (writeBytes != null && writeBytes <= 0) {
+                       Log.i(OpenP2PService.LOG_TAG, "outputStream?.write error: ");
+                       continue
+                   }
+               }
+               outputStream.close()
+// 关闭 VPN 接口
+                vpnInterface?.close()
+// 置空变量以释放资源
+                vpnInterface = null
+                Log.d(LOG_TAG, "write tun loop end")
+            }catch (e: Exception) {
+                // 捕获异常并记录
+                Log.e("VPN Connection", "发生异常: ${e.message}")
+            }
+            Log.i(OpenP2PService.LOG_TAG, "runSDWAN end");
+        }
+
     }
 
     override fun onDestroy() {
@@ -93,12 +260,13 @@ class OpenP2PService : Service() {
     }
     fun isConnected(): Boolean {
         if (!::network.isInitialized) return false
-        return network?.connect(1000)
+        return network.connect(1000)
     }
 
     fun stop() {
         running=false
         stopSelf()
+        Openp2p.stop()
     }
     @RequiresApi(Build.VERSION_CODES.O)
     private fun createNotificationChannel(channelId: String, channelName: String): String? {

app/app/src/main/java/cn/openp2p/log.kt

@@ -0,0 +1,45 @@
+package cn.openp2p
+import android.content.*
+import java.io.File
+import java.io.FileWriter
+import java.text.SimpleDateFormat
+import java.util.*
+import android.app.*
+import android.content.Context
+import android.content.Intent
+import android.graphics.Color
+import java.io.IOException
+import android.net.VpnService
+import android.os.Binder
+import android.os.Build
+import android.os.IBinder
+import android.os.ParcelFileDescriptor
+import android.util.Log
+import androidx.annotation.RequiresApi
+import androidx.core.app.NotificationCompat
+import cn.openp2p.ui.login.LoginActivity
+import kotlinx.coroutines.GlobalScope
+import kotlinx.coroutines.launch
+import openp2p.Openp2p
+import java.io.FileInputStream
+import java.io.FileOutputStream
+import java.nio.ByteBuffer
+import kotlinx.coroutines.*
+import org.json.JSONObject
+
+object Logger {
+    private val logFile: File = File("app.log")
+
+    fun log(message: String) {
+        val timestamp = SimpleDateFormat("yyyy-MM-dd HH:mm:ss", Locale.getDefault()).format(Date())
+        val logMessage = "$timestamp: $message\n"
+
+        try {
+            val fileWriter = FileWriter(logFile, true)
+            fileWriter.append(logMessage)
+            fileWriter.close()
+        } catch (e: Exception) {
+            e.printStackTrace()
+        }
+    }
+}

app/app/src/main/java/cn/openp2p/ui/login/LoginActivity.kt

@@ -5,6 +5,7 @@ import android.app.Activity
 import android.app.ActivityManager
 import android.app.Notification
 import android.app.PendingIntent
+import android.content.BroadcastReceiver
 import android.content.ComponentName
 import android.content.Context
 import android.content.Intent
@@ -25,6 +26,7 @@ import androidx.annotation.StringRes
 import androidx.appcompat.app.AppCompatActivity
 import androidx.lifecycle.Observer
 import androidx.lifecycle.ViewModelProvider
+import cn.openp2p.Logger
 import cn.openp2p.OpenP2PService
 import cn.openp2p.R
 import cn.openp2p.databinding.ActivityLoginBinding
@@ -51,6 +53,14 @@ class LoginActivity : AppCompatActivity() {
     private lateinit var loginViewModel: LoginViewModel
     private lateinit var binding: ActivityLoginBinding
     private lateinit var mService: OpenP2PService
+    @RequiresApi(Build.VERSION_CODES.O)
+    override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) {
+        super.onActivityResult(requestCode, resultCode, data)
+        if (requestCode == 0 && resultCode == Activity.RESULT_OK) {
+            startService(Intent(this, OpenP2PService::class.java))
+        }
+    }
+
     @RequiresApi(Build.VERSION_CODES.O)
     override fun onCreate(savedInstanceState: Bundle?) {
         super.onCreate(savedInstanceState)
@@ -78,22 +88,18 @@ class LoginActivity : AppCompatActivity() {
                 token.error = getString(loginState.passwordError)
             }
         })
-        val intent1 = VpnService.prepare(this) ?: return
+        openp2pLog.setText(R.string.phone_setting)
-        loginViewModel.loginResult.observe(this@LoginActivity, Observer {
+        val intent = VpnService.prepare(this)
-            val loginResult = it ?: return@Observer
+         if (intent != null)
+         {
+             Log.i("openp2p", "VpnService.prepare need permission");
+             startActivityForResult(intent, 0)
+         }
+         else {
+             Log.i("openp2p", "VpnService.prepare ready");
+             onActivityResult(0, Activity.RESULT_OK, null)
+         }
 
-            loading.visibility = View.GONE
-            if (loginResult.error != null) {
-                showLoginFailed(loginResult.error)
-            }
-            if (loginResult.success != null) {
-                updateUiWithUser(loginResult.success)
-            }
-            setResult(Activity.RESULT_OK)
-
-            //Complete and destroy login activity once successful
-            finish()
-        })
 
         profile.setOnClickListener {
             val url = "https://console.openp2p.cn/profile"
@@ -110,17 +116,23 @@ class LoginActivity : AppCompatActivity() {
             }
 
             openp2pLog.setText(R.string.phone_setting)
-            token.setText(Openp2p.getToken(getExternalFilesDir(null).toString()))
+
             login.setOnClickListener {
                 if (login.text.toString()=="退出"){
-//                    val intent = Intent(this@LoginActivity, OpenP2PService::class.java)
+                //    val intent = Intent(this@LoginActivity, OpenP2PService::class.java)
-//                    stopService(intent)
+                //    stopService(intent)
                     Log.i(LOG_TAG, "quit")
                     mService.stop()
-                    unbindService(connection)
+
                     val intent = Intent(this@LoginActivity, OpenP2PService::class.java)
                     stopService(intent)
+                    // 解绑服务
+                    unbindService(connection)
+
+                    // 结束当前 Activity
+                    finish() // 或者使用 finishAffinity() 来结束整个应用程序
                     exitAPP()
+                    // finishAffinity()
 
                 }
                 login.setText("退出")
@@ -139,13 +151,20 @@ class LoginActivity : AppCompatActivity() {
                             if (isConnect) {
                                 onlineState.setText("在线")
                             } else {
-                                onlineState.setText("离线")
+                                onlineState.setText("正在登录")
                             }
                         }
                     } while (true)
                 }
 
             }
+            val tokenText = Openp2p.getToken(getExternalFilesDir(null).toString())
+            token.setText(tokenText.toString())
+            // Check token length and automatically click login if length > 10
+            if (tokenText.length > 10) {
+//                Logger.log("performClick ")
+                login.performClick()
+            }
         }
     }
     @RequiresApi(Build.VERSION_CODES.LOLLIPOP)

app/app/src/main/res/layout/activity_login.xml

@@ -13,28 +13,27 @@
 
     <EditText
         android:id="@+id/token"
-        android:layout_width="225dp"
+        android:layout_width="250dp"
-        android:layout_height="46dp"
+        android:layout_height="45dp"
         android:hint="Token"
         android:imeActionLabel="@string/action_sign_in_short"
         android:imeOptions="actionDone"
         android:selectAllOnFocus="true"
         app:layout_constraintStart_toStartOf="parent"
-        app:layout_constraintTop_toTopOf="parent"
+        app:layout_constraintTop_toTopOf="parent" />
-         />
 
     <Button
         android:id="@+id/login"
-        android:layout_width="wrap_content"
+        android:layout_width="85dp"
-        android:layout_height="wrap_content"
+        android:layout_height="45dp"
         android:layout_gravity="start"
         android:layout_marginStart="24dp"
         android:layout_marginLeft="24dp"
         android:enabled="false"
         android:text="@string/action_sign_in"
+        app:layout_constraintEnd_toEndOf="parent"
         app:layout_constraintStart_toEndOf="@+id/token"
-        app:layout_constraintTop_toTopOf="parent"
+        app:layout_constraintTop_toTopOf="parent" />
-        tools:layout_editor_absoluteY="-2dp" />
 
 
 
@@ -58,28 +57,36 @@
         android:id="@+id/openp2pLog"
         android:layout_width="359dp"
         android:layout_height="548dp"
+        android:layout_marginTop="24dp"
         android:ems="10"
-        android:inputType="none"
-        android:textIsSelectable="true"
         android:focusable="false"
+        android:inputType="none"
         android:text="Name"
+        android:textIsSelectable="true"
+        app:layout_constraintEnd_toEndOf="parent"
         app:layout_constraintStart_toStartOf="parent"
         app:layout_constraintTop_toBottomOf="@+id/onlineState" />
+
     <Button
         android:id="@+id/profile"
-        android:layout_width="wrap_content"
+        android:layout_width="250dp"
-        android:layout_height="wrap_content"
+        android:layout_height="45dp"
+        android:layout_marginTop="8dp"
         android:text="打开控制台查看Token"
         app:layout_constraintStart_toStartOf="parent"
         app:layout_constraintTop_toBottomOf="@+id/token" />
 
     <EditText
         android:id="@+id/onlineState"
-        android:layout_width="113dp"
+        android:layout_width="85dp"
         android:layout_height="45dp"
+        android:layout_marginStart="24dp"
+        android:layout_marginLeft="24dp"
+        android:layout_marginTop="8dp"
         android:ems="10"
         android:inputType="textPersonName"
         android:text="未登录"
+        app:layout_constraintEnd_toEndOf="parent"
         app:layout_constraintStart_toEndOf="@+id/profile"
         app:layout_constraintTop_toBottomOf="@+id/login" />
 

app/app/src/main/res/values/strings.xml

@@ -10,7 +10,14 @@
     <string name="invalid_password">Token可以在 https://console.openp2p.cn/profile 获得</string>
     <string name="login_failed">"Login failed"</string>
     <string name="phone_setting">"安卓系统默认设置的”杀后台进程“会导致 OpenP2P 在后台运行一会后，被系统杀死进程，导致您的体验受到影响。您可以通过以下方式修改几个设置，解决此问题：
-
+华为鸿蒙：
+    1. 允许应用后台运行：进入设置 → 搜索进入 应用启动管理 → 关闭 OpenP2P 的 自动管理 开关 → 在弹框中勾选 允许后台活动 
+    2. 避免应用被电池优化程序清理：进入设置 → 搜索进入电池优化 → 不允许 →选择所有应用 → 找到无法后台运行的应用 → 设置为不允许
+    3. 关闭省电模式：进入设置 → 电池 → 关闭 省电模式 开关
+    4. 保持设备网络连接：进入设置  → 电池 → 更多电池设置 → 开启 休眠时始终保持网络连接 开关。
+    5. 给后台运行的应用加锁：打开应用后 → 进入多任务界面 → 下拉选中的卡片进行加锁 → 然后点击清理图标清理其他不经常使用的应用
+    6. 设置开发人员选项中相关开关：进入设置 → 搜索进入 开发人员选项 → 找到 不保留活动 开关后关闭 → 并在 后台进程限制 选择 标准限制
+    
 华为手机：
     进入”设置“，搜索并进入“电池优化“界面，选中 OpenP2P 程序，不允许系统对其进行电池优化；
     进入”设置“，进入”应用管理“界面，选中 OpenP2P 程序，点击”耗电情况“，开启”允许后台活动“即可；

app/build.gradle

@@ -0,0 +1,30 @@
+// Top-level build file where you can add configuration options common to all sub-projects/modules.
+
+buildscript {
+
+    ext.kotlin_version = "1.8.20"
+    repositories {
+        google()
+        mavenCentral()
+    }
+    dependencies {
+        classpath "com.android.tools.build:gradle:8.1.3"
+        classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
+
+        // NOTE: Do not place your application dependencies here; they belong
+        // in the individual module build.gradle files
+    }
+}
+
+
+allprojects {
+    repositories {
+        google()
+        mavenCentral()
+        jcenter() // Warning: this repository is going to shut down soon
+    }
+}
+
+task clean(type: Delete) {
+    delete rootProject.buildDir
+}

app/gradle/wrapper/gradle-wrapper.properties

@@ -1,6 +1,6 @@
-#Sat Oct 22 21:46:24 CST 2022
+#Tue Dec 05 16:04:08 CST 2023
 distributionBase=GRADLE_USER_HOME
-distributionUrl=https\://services.gradle.org/distributions/gradle-6.7.1-bin.zip
 distributionPath=wrapper/dists
-zipStorePath=wrapper/dists
+distributionUrl=https\://services.gradle.org/distributions/gradle-8.2-bin.zip
 zipStoreBase=GRADLE_USER_HOME
+zipStorePath=wrapper/dists

cmd/openp2p.go

@@ -1,7 +1,9 @@
-package main
+package main
-
+
-import openp2p "openp2p/core"
+import (
-
+	op "openp2p/core"
-func main() {
+)
-	openp2p.Run()
+
-}
+func main() {
+	op.Run()
+}

core/bandwidthLimit.go

@@ -1,45 +0,0 @@
-package openp2p
-
-import (
-	"sync"
-	"time"
-)
-
-// BandwidthLimiter ...
-type BandwidthLimiter struct {
-	ts           time.Time
-	bw           int // mbps
-	freeBytes    int // bytes
-	maxFreeBytes int // bytes
-	mtx          sync.Mutex
-}
-
-// mbps
-func newBandwidthLimiter(bw int) *BandwidthLimiter {
-	return &BandwidthLimiter{
-		bw:           bw,
-		ts:           time.Now(),
-		maxFreeBytes: bw * 1024 * 1024 / 8,
-		freeBytes:    bw * 1024 * 1024 / 8,
-	}
-}
-
-// Add ...
-func (bl *BandwidthLimiter) Add(bytes int) {
-	if bl.bw <= 0 {
-		return
-	}
-	bl.mtx.Lock()
-	defer bl.mtx.Unlock()
-	// calc free flow 1000*1000/1024/1024=0.954; 1024*1024/1000/1000=1.048
-	bl.freeBytes += int(time.Since(bl.ts) * time.Duration(bl.bw) / 8 / 954)
-	if bl.freeBytes > bl.maxFreeBytes {
-		bl.freeBytes = bl.maxFreeBytes
-	}
-	bl.freeBytes -= bytes
-	bl.ts = time.Now()
-	if bl.freeBytes < 0 {
-		// sleep for the overflow
-		time.Sleep(time.Millisecond * time.Duration(-bl.freeBytes/(bl.bw*1048/8)))
-	}
-}

core/common.go

@@ -5,8 +5,10 @@ import (
 	"crypto/aes"
 	"crypto/cipher"
 	"crypto/tls"
+	"encoding/binary"
 	"encoding/json"
 	"fmt"
+	"math/big"
 	"math/rand"
 	"net"
 	"net/http"
@@ -137,8 +139,7 @@ func netInfo() *NetInfo {
 			continue
 		}
 		rsp := NetInfo{}
-		err = json.Unmarshal(buf[:n], &rsp)
+		if err = json.Unmarshal(buf[:n], &rsp); err != nil {
-		if err != nil {
 			gLog.Printf(LvERROR, "wrong NetInfo:%s", err)
 			continue
 		}
@@ -198,8 +199,12 @@ func parseMajorVer(ver string) int {
 	return 0
 }
 
-func IsIPv6(address string) bool {
+func IsIPv6(ipStr string) bool {
-	return strings.Count(address, ":") >= 2
+	ip := net.ParseIP(ipStr)
+	if ip == nil {
+		return false
+	}
+	return ip.To16() != nil && ip.To4() == nil
 }
 
 var letters = []byte("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890-")
@@ -211,3 +216,67 @@ func randStr(n int) string {
 	}
 	return string(b)
 }
+
+func execCommand(commandPath string, wait bool, arg ...string) (err error) {
+	command := exec.Command(commandPath, arg...)
+	err = command.Start()
+	if err != nil {
+		return
+	}
+	if wait {
+		err = command.Wait()
+	}
+	return
+}
+
+func sanitizeFileName(fileName string) string {
+	validFileName := fileName
+	invalidChars := []string{"\\", "/", ":", "*", "?", "\"", "<", ">", "|"}
+	for _, char := range invalidChars {
+		validFileName = strings.ReplaceAll(validFileName, char, " ")
+	}
+	return validFileName
+}
+
+func prettyJson(s interface{}) string {
+	jsonData, err := json.MarshalIndent(s, "", "  ")
+	if err != nil {
+		fmt.Println("Error marshalling JSON:", err)
+		return ""
+	}
+	return string(jsonData)
+}
+
+func inetAtoN(ipstr string) (uint32, error) { // support both ipnet or single ip
+	i, _, err := net.ParseCIDR(ipstr)
+	if err != nil {
+		i = net.ParseIP(ipstr)
+		if i == nil {
+			return 0, err
+		}
+	}
+	ret := big.NewInt(0)
+	ret.SetBytes(i.To4())
+	return uint32(ret.Int64()), nil
+}
+
+func calculateChecksum(data []byte) uint16 {
+	length := len(data)
+	sum := uint32(0)
+
+	// Calculate the sum of 16-bit words
+	for i := 0; i < length-1; i += 2 {
+		sum += uint32(binary.BigEndian.Uint16(data[i : i+2]))
+	}
+
+	// Add the last byte (if odd length)
+	if length%2 != 0 {
+		sum += uint32(data[length-1])
+	}
+
+	// Fold 32-bit sum to 16 bits
+	sum = (sum >> 16) + (sum & 0xffff)
+	sum += (sum >> 16)
+
+	return uint16(^sum)
+}

core/common_test.go

@@ -94,3 +94,23 @@ func TestParseMajorVer(t *testing.T) {
 	assertParseMajorVer(t, "3.0.0", 3)
 
 }
+
+func TestIsIPv6(t *testing.T) {
+	tests := []struct {
+		ipStr string
+		want  bool
+	}{
+		{"2001:0db8:85a3:0000:0000:8a2e:0370:7334", true}, // 有效的 IPv6 地址
+		{"2001:db8::2:1", true},                           // 有效的 IPv6 地址
+		{"192.168.1.1", false},                            // 无效的 IPv6 地址，是 IPv4
+		{"2001:db8::G:1", false},                          // 无效的 IPv6 地址，包含非法字符
+		// 可以添加更多测试用例
+	}
+
+	for _, tt := range tests {
+		got := IsIPv6(tt.ipStr)
+		if got != tt.want {
+			t.Errorf("isValidIPv6(%s) = %v, want %v", tt.ipStr, got, tt.want)
+		}
+	}
+}

core/config.go

@@ -3,8 +3,10 @@ package openp2p
 import (
 	"encoding/json"
 	"flag"
-	"io/ioutil"
+	"fmt"
 	"os"
+	"strconv"
+	"strings"
 	"sync"
 	"time"
 )
@@ -13,18 +15,25 @@ var gConf Config
 
 type AppConfig struct {
 	// required
-	AppName  string
+	AppName          string
-	Protocol string
+	Protocol         string
-	SrcPort  int
+	UnderlayProtocol string
-	PeerNode string
+	PunchPriority    int // bitwise DisableTCP|DisableUDP|TCPFirst  0:tcp and udp both enable, udp first
-	DstPort  int
+	Whitelist        string
-	DstHost  string
+	SrcPort          int
-	PeerUser string
+	PeerNode         string
-	Enabled  int // default:1
+	DstPort          int
+	DstHost          string
+	PeerUser         string
+	RelayNode        string
+	ForceRelay       int // default:0 disable;1 enable
+	Enabled          int // default:1
 	// runtime info
+	relayMode        string // private|public
 	peerVersion      string
 	peerToken        uint64
 	peerNatType      int
+	peerLanIP        string
 	hasIPv4          int
 	peerIPv6         string
 	hasUPNPorNATPMP  int
@@ -38,16 +47,95 @@ type AppConfig struct {
 	connectTime      time.Time
 	fromToken        uint64
 	linkMode         string
-	isUnderlayServer int // TODO: bool?
+	isUnderlayServer int
+}
+
+const (
+	PunchPriorityTCPFirst   = 1
+	PunchPriorityUDPDisable = 1 << 1
+	PunchPriorityTCPDisable = 1 << 2
+)
+
+func (c *AppConfig) ID() uint64 {
+	if c.SrcPort == 0 { // memapp
+		return NodeNameToID(c.PeerNode)
+	}
+	if c.Protocol == "tcp" {
+		return uint64(c.SrcPort) * 10
+	}
+	return uint64(c.SrcPort)*10 + 1
+}
+
+func (c *AppConfig) LogPeerNode() string {
+	if c.relayMode == "public" { // memapp
+		return fmt.Sprintf("%d", NodeNameToID(c.PeerNode))
+	}
+	return c.PeerNode
 }
 
-// TODO: add loglevel, maxlogfilesize
 type Config struct {
-	Network    NetworkConfig `json:"network"`
+	Network NetworkConfig `json:"network"`
-	Apps       []*AppConfig  `json:"apps"`
+	Apps    []*AppConfig  `json:"apps"`
+
 	LogLevel   int
 	daemonMode bool
 	mtx        sync.Mutex
+	sdwanMtx   sync.Mutex
+	sdwan      SDWANInfo
+	delNodes   []SDWANNode
+	addNodes   []SDWANNode
+}
+
+func (c *Config) getSDWAN() SDWANInfo {
+	c.sdwanMtx.Lock()
+	defer c.sdwanMtx.Unlock()
+	return c.sdwan
+}
+
+func (c *Config) getDelNodes() []SDWANNode {
+	c.sdwanMtx.Lock()
+	defer c.sdwanMtx.Unlock()
+	return c.delNodes
+}
+
+func (c *Config) getAddNodes() []SDWANNode {
+	c.sdwanMtx.Lock()
+	defer c.sdwanMtx.Unlock()
+	return c.addNodes
+}
+
+func (c *Config) setSDWAN(s SDWANInfo) {
+	c.sdwanMtx.Lock()
+	defer c.sdwanMtx.Unlock()
+	// get old-new
+	c.delNodes = []SDWANNode{}
+	for _, oldNode := range c.sdwan.Nodes {
+		isDeleted := true
+		for _, newNode := range s.Nodes {
+			if oldNode.Name == newNode.Name && oldNode.IP == newNode.IP && oldNode.Resource == newNode.Resource && c.sdwan.Mode == s.Mode && c.sdwan.CentralNode == s.CentralNode {
+				isDeleted = false
+				break
+			}
+		}
+		if isDeleted {
+			c.delNodes = append(c.delNodes, oldNode)
+		}
+	}
+	// get new-old
+	c.addNodes = []SDWANNode{}
+	for _, newNode := range s.Nodes {
+		isNew := true
+		for _, oldNode := range c.sdwan.Nodes {
+			if oldNode.Name == newNode.Name && oldNode.IP == newNode.IP && oldNode.Resource == newNode.Resource && c.sdwan.Mode == s.Mode && c.sdwan.CentralNode == s.CentralNode {
+				isNew = false
+				break
+			}
+		}
+		if isNew {
+			c.addNodes = append(c.addNodes, newNode)
+		}
+	}
+	c.sdwan = s
 }
 
 func (c *Config) switchApp(app AppConfig, enabled int) {
@@ -58,21 +146,77 @@ func (c *Config) switchApp(app AppConfig, enabled int) {
 			c.Apps[i].Enabled = enabled
 			c.Apps[i].retryNum = 0
 			c.Apps[i].nextRetryTime = time.Now()
-			return
+			break
 		}
 	}
+	c.save()
+}
+
+func (c *Config) retryApp(peerNode string) {
+	GNetwork.apps.Range(func(id, i interface{}) bool {
+		app := i.(*p2pApp)
+		if app.config.PeerNode == peerNode {
+			gLog.Println(LvDEBUG, "retry app ", app.config.LogPeerNode())
+			app.config.retryNum = 0
+			app.config.nextRetryTime = time.Now()
+			app.retryRelayNum = 0
+			app.nextRetryRelayTime = time.Now()
+			app.hbMtx.Lock()
+			app.hbTimeRelay = time.Now().Add(-TunnelHeartbeatTime * 3)
+			app.hbMtx.Unlock()
+		}
+		if app.config.RelayNode == peerNode {
+			gLog.Println(LvDEBUG, "retry app ", app.config.LogPeerNode())
+			app.retryRelayNum = 0
+			app.nextRetryRelayTime = time.Now()
+			app.hbMtx.Lock()
+			app.hbTimeRelay = time.Now().Add(-TunnelHeartbeatTime * 3)
+			app.hbMtx.Unlock()
+		}
+		return true
+	})
+}
+
+func (c *Config) retryAllApp() {
+	GNetwork.apps.Range(func(id, i interface{}) bool {
+		app := i.(*p2pApp)
+		gLog.Println(LvDEBUG, "retry app ", app.config.LogPeerNode())
+		app.config.retryNum = 0
+		app.config.nextRetryTime = time.Now()
+		app.retryRelayNum = 0
+		app.nextRetryRelayTime = time.Now()
+		app.hbMtx.Lock()
+		defer app.hbMtx.Unlock()
+		app.hbTimeRelay = time.Now().Add(-TunnelHeartbeatTime * 3)
+		return true
+	})
+}
+
+func (c *Config) retryAllMemApp() {
+	GNetwork.apps.Range(func(id, i interface{}) bool {
+		app := i.(*p2pApp)
+		if app.config.SrcPort != 0 {
+			return true
+		}
+		gLog.Println(LvDEBUG, "retry app ", app.config.LogPeerNode())
+		app.config.retryNum = 0
+		app.config.nextRetryTime = time.Now()
+		app.retryRelayNum = 0
+		app.nextRetryRelayTime = time.Now()
+		app.hbMtx.Lock()
+		defer app.hbMtx.Unlock()
+		app.hbTimeRelay = time.Now().Add(-TunnelHeartbeatTime * 3)
+		return true
+	})
 }
 
 func (c *Config) add(app AppConfig, override bool) {
 	c.mtx.Lock()
 	defer c.mtx.Unlock()
-	if app.SrcPort == 0 || app.DstPort == 0 {
+	defer c.save()
-		gLog.Println(LvERROR, "invalid app ", app)
-		return
-	}
 	if override {
 		for i := 0; i < len(c.Apps); i++ {
-			if c.Apps[i].Protocol == app.Protocol && c.Apps[i].SrcPort == app.SrcPort {
+			if c.Apps[i].PeerNode == app.PeerNode && c.Apps[i].Protocol == app.Protocol && c.Apps[i].SrcPort == app.SrcPort {
 				c.Apps[i] = &app // override it
 				return
 			}
@@ -82,31 +226,60 @@ func (c *Config) add(app AppConfig, override bool) {
 }
 
 func (c *Config) delete(app AppConfig) {
-	if app.SrcPort == 0 || app.DstPort == 0 {
-		return
-	}
 	c.mtx.Lock()
 	defer c.mtx.Unlock()
+	defer c.save()
 	for i := 0; i < len(c.Apps); i++ {
-		if c.Apps[i].Protocol == app.Protocol && c.Apps[i].SrcPort == app.SrcPort {
+		got := false
-			c.Apps = append(c.Apps[:i], c.Apps[i+1:]...)
+		if app.SrcPort != 0 { // normal p2papp
+			if c.Apps[i].Protocol == app.Protocol && c.Apps[i].SrcPort == app.SrcPort {
+				got = true
+			}
+		} else { // memapp
+			if c.Apps[i].PeerNode == app.PeerNode {
+				got = true
+			}
+		}
+		if got {
+			if i == len(c.Apps)-1 {
+				c.Apps = c.Apps[:i]
+			} else {
+				c.Apps = append(c.Apps[:i], c.Apps[i+1:]...)
+			}
 			return
 		}
 	}
+
 }
 
 func (c *Config) save() {
-	c.mtx.Lock()
+	// c.mtx.Lock()
-	defer c.mtx.Unlock()
+	// defer c.mtx.Unlock()  // internal call
+	if c.Network.Token == 0 {
+		return
+	}
 	data, _ := json.MarshalIndent(c, "", "  ")
-	err := ioutil.WriteFile("config.json", data, 0644)
+	err := os.WriteFile("config.json", data, 0644)
 	if err != nil {
 		gLog.Println(LvERROR, "save config.json error:", err)
 	}
 }
 
+func (c *Config) saveCache() {
+	// c.mtx.Lock()
+	// defer c.mtx.Unlock()  // internal call
+	if c.Network.Token == 0 {
+		return
+	}
+	data, _ := json.MarshalIndent(c, "", "  ")
+	err := os.WriteFile("config.json0", data, 0644)
+	if err != nil {
+		gLog.Println(LvERROR, "save config.json0 error:", err)
+	}
+}
+
 func init() {
-	gConf.LogLevel = 1
+	gConf.LogLevel = int(LvINFO)
 	gConf.Network.ShareBandwidth = 10
 	gConf.Network.ServerHost = "api.openp2p.cn"
 	gConf.Network.ServerPort = WsPort
@@ -116,43 +289,92 @@ func init() {
 func (c *Config) load() error {
 	c.mtx.Lock()
 	defer c.mtx.Unlock()
-	data, err := ioutil.ReadFile("config.json")
+	data, err := os.ReadFile("config.json")
 	if err != nil {
-		// gLog.Println(LevelERROR, "read config.json error:", err)
+		return c.loadCache()
-		return err
 	}
 	err = json.Unmarshal(data, &c)
 	if err != nil {
 		gLog.Println(LvERROR, "parse config.json error:", err)
+		// try cache
+		return c.loadCache()
+	}
+	// load ok. cache it
+	var filteredApps []*AppConfig // filter memapp
+	for _, app := range c.Apps {
+		if app.SrcPort != 0 {
+			filteredApps = append(filteredApps, app)
+		}
+	}
+	c.Apps = filteredApps
+	c.saveCache()
+	return err
+}
+
+func (c *Config) loadCache() error {
+	data, err := os.ReadFile("config.json0")
+	if err != nil {
+		return err
+	}
+	err = json.Unmarshal(data, &c)
+	if err != nil {
+		gLog.Println(LvERROR, "parse config.json0 error:", err)
 	}
 	return err
 }
 
+// deal with multi-thread r/w
 func (c *Config) setToken(token uint64) {
 	c.mtx.Lock()
 	defer c.mtx.Unlock()
-	c.Network.Token = token
+	defer c.save()
+	if token != 0 {
+		c.Network.Token = token
+	}
 }
 func (c *Config) setUser(user string) {
 	c.mtx.Lock()
 	defer c.mtx.Unlock()
+	defer c.save()
 	c.Network.User = user
 }
 func (c *Config) setNode(node string) {
 	c.mtx.Lock()
 	defer c.mtx.Unlock()
+	defer c.save()
 	c.Network.Node = node
+	c.Network.nodeID = NodeNameToID(c.Network.Node)
+}
+func (c *Config) nodeID() uint64 {
+	c.mtx.Lock()
+	defer c.mtx.Unlock()
+	if c.Network.nodeID == 0 {
+		c.Network.nodeID = NodeNameToID(c.Network.Node)
+	}
+	return c.Network.nodeID
 }
 func (c *Config) setShareBandwidth(bw int) {
 	c.mtx.Lock()
 	defer c.mtx.Unlock()
+	defer c.save()
 	c.Network.ShareBandwidth = bw
 }
+func (c *Config) setIPv6(v6 string) {
+	c.mtx.Lock()
+	defer c.mtx.Unlock()
+	c.Network.publicIPv6 = v6
+}
+func (c *Config) IPv6() string {
+	c.mtx.Lock()
+	defer c.mtx.Unlock()
+	return c.Network.publicIPv6
+}
 
 type NetworkConfig struct {
 	// local info
 	Token           uint64
 	Node            string
+	nodeID          uint64
 	User            string
 	localIP         string
 	mac             string
@@ -171,7 +393,7 @@ type NetworkConfig struct {
 	TCPPort    int
 }
 
-func parseParams(subCommand string) {
+func parseParams(subCommand string, cmd string) {
 	fset := flag.NewFlagSet(subCommand, flag.ExitOnError)
 	serverHost := fset.String("serverhost", "api.openp2p.cn", "server host ")
 	serverPort := fset.Int("serverport", WsPort, "server port ")
@@ -180,33 +402,49 @@ func parseParams(subCommand string) {
 	node := fset.String("node", "", "node name. 8-31 characters. if not set, it will be hostname")
 	peerNode := fset.String("peernode", "", "peer node name that you want to connect")
 	dstIP := fset.String("dstip", "127.0.0.1", "destination ip ")
+	whiteList := fset.String("whitelist", "", "whitelist for p2pApp ")
 	dstPort := fset.Int("dstport", 0, "destination port ")
 	srcPort := fset.Int("srcport", 0, "source port ")
 	tcpPort := fset.Int("tcpport", 0, "tcp port for upnp or publicip")
 	protocol := fset.String("protocol", "tcp", "tcp or udp")
+	underlayProtocol := fset.String("underlay_protocol", "quic", "quic or kcp")
+	punchPriority := fset.Int("punch_priority", 0, "bitwise DisableTCP|DisableUDP|UDPFirst  0:tcp and udp both enable, tcp first")
 	appName := fset.String("appname", "", "app name")
+	relayNode := fset.String("relaynode", "", "relaynode")
 	shareBandwidth := fset.Int("sharebandwidth", 10, "N mbps share bandwidth limit, private network no limit")
 	daemonMode := fset.Bool("d", false, "daemonMode")
 	notVerbose := fset.Bool("nv", false, "not log console")
 	newconfig := fset.Bool("newconfig", false, "not load existing config.json")
-	logLevel := fset.Int("loglevel", 0, "0:info 1:warn 2:error 3:debug")
+	logLevel := fset.Int("loglevel", 1, "0:debug 1:info 2:warn 3:error")
-	if subCommand == "" { // no subcommand
+	maxLogSize := fset.Int("maxlogsize", 1024*1024, "default 1MB")
-		fset.Parse(os.Args[1:])
+	if cmd == "" {
+		if subCommand == "" { // no subcommand
+			fset.Parse(os.Args[1:])
+		} else {
+			fset.Parse(os.Args[2:])
+		}
 	} else {
-		fset.Parse(os.Args[2:])
+		gLog.Println(LvINFO, "cmd=", cmd)
+		args := strings.Split(cmd, " ")
+		fset.Parse(args)
 	}
 
+	gLog.setMaxSize(int64(*maxLogSize))
 	config := AppConfig{Enabled: 1}
 	config.PeerNode = *peerNode
 	config.DstHost = *dstIP
+	config.Whitelist = *whiteList
 	config.DstPort = *dstPort
 	config.SrcPort = *srcPort
 	config.Protocol = *protocol
+	config.UnderlayProtocol = *underlayProtocol
+	config.PunchPriority = *punchPriority
 	config.AppName = *appName
+	config.RelayNode = *relayNode
 	if !*newconfig {
 		gConf.load() // load old config. otherwise will clear all apps
 	}
-	if config.SrcPort != 0 {
+	if config.SrcPort != 0 { // filter memapp
 		gConf.add(config, true)
 	}
 	// gConf.mtx.Lock() // when calling this func it's single-thread no lock
@@ -217,7 +455,7 @@ func parseParams(subCommand string) {
 			gConf.Network.ShareBandwidth = *shareBandwidth
 		}
 		if f.Name == "node" {
-			gConf.Network.Node = *node
+			gConf.setNode(*node)
 		}
 		if f.Name == "serverhost" {
 			gConf.Network.ServerHost = *serverHost
@@ -229,32 +467,39 @@ func parseParams(subCommand string) {
 			gConf.Network.TCPPort = *tcpPort
 		}
 		if f.Name == "token" {
-			gConf.Network.Token = *token
+			gConf.setToken(*token)
 		}
 	})
-
+	// set default value
 	if gConf.Network.ServerHost == "" {
 		gConf.Network.ServerHost = *serverHost
 	}
 	if *node != "" {
-		if len(*node) < MinNodeNameLen {
+		gConf.setNode(*node)
-			gLog.Println(LvERROR, ErrNodeTooShort)
-			os.Exit(9)
-		}
-		gConf.Network.Node = *node
 	} else {
+		envNode := os.Getenv("OPENP2P_NODE")
+		if envNode != "" {
+			gConf.setNode(envNode)
+		}
 		if gConf.Network.Node == "" { // if node name not set. use os.Hostname
-			gConf.Network.Node = defaultNodeName()
+			gConf.setNode(defaultNodeName())
 		}
 	}
 	if gConf.Network.TCPPort == 0 {
 		if *tcpPort == 0 {
-			p := int(nodeNameToID(gConf.Network.Node)%15000 + 50000)
+			p := int(gConf.nodeID()%15000 + 50000)
 			tcpPort = &p
 		}
 		gConf.Network.TCPPort = *tcpPort
 	}
-
+	if *token == 0 {
+		envToken := os.Getenv("OPENP2P_TOKEN")
+		if envToken != "" {
+			if n, err := strconv.ParseUint(envToken, 10, 64); n != 0 && err == nil {
+				gConf.setToken(n)
+			}
+		}
+	}
 	gConf.Network.ServerPort = *serverPort
 	gConf.Network.UDPPort1 = UDPPort1
 	gConf.Network.UDPPort2 = UDPPort2

core/config_test.go

@@ -0,0 +1,178 @@
+package openp2p
+
+import (
+	"encoding/json"
+	"testing"
+)
+
+func TestSetSDWAN_ChangeNode(t *testing.T) {
+	conf := Config{}
+	sdwanInfo := SDWANInfo{}
+	sdwanStr := `{"id":1312667996276071700,"name":"network1","gateway":"10.2.3.254/24","mode":"fullmesh","centralNode":"n1-stable","enable":1,"Nodes":[{"name":"222-debug","ip":"10.2.3.13"},{"name":"222stable","ip":"10.2.3.222"},{"name":"5800-debug","ip":"10.2.3.56"},{"name":"Mate60pro","ip":"10.2.3.60"},{"name":"Mymatepad2023","ip":"10.2.3.23"},{"name":"n1-stable","ip":"10.2.3.29","resource":"192.168.3.0/24"},{"name":"tony-stable","ip":"10.2.3.4","resource":"10.1.0.0/16"}]}`
+	if err := json.Unmarshal([]byte(sdwanStr), &sdwanInfo); err != nil {
+		t.Errorf("unmarshal error")
+		return
+	}
+
+	conf.setSDWAN(sdwanInfo)
+	if len(conf.getDelNodes()) > 0 {
+		t.Errorf("getDelNodes error")
+		return
+	}
+	if len(conf.getAddNodes()) != 7 {
+		t.Errorf("getAddNodes error")
+		return
+	}
+	sdwanInfo2 := SDWANInfo{}
+	sdwanStr = `{"id":1312667996276071700,"name":"network1","gateway":"10.2.3.254/24","mode":"fullmesh","centralNode":"n1-stable","enable":1,"Nodes":[{"name":"222-debug","ip":"10.2.3.13"},{"name":"222stable","ip":"10.2.3.222"},{"name":"5800-debug","ip":"10.2.3.56"},{"name":"Mate60pro","ip":"10.2.3.60"},{"name":"Mymatepad2023","ip":"10.2.3.23"},{"name":"n1-stable","ip":"10.2.3.29","resource":"192.168.3.0/24"}]}`
+	if err := json.Unmarshal([]byte(sdwanStr), &sdwanInfo2); err != nil {
+		t.Errorf("unmarshal error")
+		return
+	}
+
+	conf.setSDWAN(sdwanInfo2)
+	diff := conf.getDelNodes()
+	if len(diff) != 1 && diff[0].IP != "10.2.3.4" {
+		t.Errorf("getDelNodes error")
+		return
+	}
+	sdwanInfo3 := SDWANInfo{}
+	sdwanStr = `{"id":1312667996276071700,"name":"network1","gateway":"10.2.3.254/24","mode":"fullmesh","centralNode":"n1-stable","enable":1,"Nodes":[{"name":"222-debug","ip":"10.2.3.13"},{"name":"222stable","ip":"10.2.3.222"},{"name":"5800-debug","ip":"10.2.3.56"},{"name":"Mymatepad2023","ip":"10.2.3.23"},{"name":"n1-stable","ip":"10.2.3.29","resource":"192.168.3.0/24"}]}`
+	if err := json.Unmarshal([]byte(sdwanStr), &sdwanInfo3); err != nil {
+		t.Errorf("unmarshal error")
+		return
+	}
+
+	conf.setSDWAN(sdwanInfo3)
+	diff = conf.getDelNodes()
+	if len(diff) != 1 && diff[0].IP != "10.2.3.60" {
+		t.Errorf("getDelNodes error")
+		return
+	}
+	// add new node
+	sdwanInfo4 := SDWANInfo{}
+	sdwanStr = `{"id":1312667996276071700,"name":"network1","gateway":"10.2.3.254/24","mode":"fullmesh","centralNode":"n1-stable","enable":1,"Nodes":[{"name":"222-debug","ip":"10.2.3.13"},{"name":"222stable","ip":"10.2.3.222"},{"name":"5800-debug","ip":"10.2.3.56"},{"name":"Mate60pro","ip":"10.2.3.60"},{"name":"Mymatepad2023","ip":"10.2.3.23"},{"name":"n1-stable","ip":"10.2.3.29","resource":"192.168.3.0/24"}]}`
+	if err := json.Unmarshal([]byte(sdwanStr), &sdwanInfo4); err != nil {
+		t.Errorf("unmarshal error")
+		return
+	}
+
+	conf.setSDWAN(sdwanInfo4)
+	diff = conf.getDelNodes()
+	if len(diff) > 0 {
+		t.Errorf("getDelNodes error")
+		return
+	}
+	diff = conf.getAddNodes()
+	if len(diff) != 1 && diff[0].IP != "10.2.3.60" {
+		t.Errorf("getAddNodes error")
+		return
+	}
+}
+
+func TestSetSDWAN_ChangeNodeIP(t *testing.T) {
+	conf := Config{}
+	sdwanInfo := SDWANInfo{}
+	sdwanStr := `{"id":1312667996276071700,"name":"network1","gateway":"10.2.3.254/24","mode":"fullmesh","centralNode":"n1-stable","enable":1,"Nodes":[{"name":"222-debug","ip":"10.2.3.13"},{"name":"222stable","ip":"10.2.3.222"},{"name":"5800-debug","ip":"10.2.3.56"},{"name":"Mate60pro","ip":"10.2.3.60"},{"name":"Mymatepad2023","ip":"10.2.3.23"},{"name":"n1-stable","ip":"10.2.3.29","resource":"192.168.3.0/24"},{"name":"tony-stable","ip":"10.2.3.4","resource":"10.1.0.0/16"}]}`
+	if err := json.Unmarshal([]byte(sdwanStr), &sdwanInfo); err != nil {
+		t.Errorf("unmarshal error")
+		return
+	}
+
+	conf.setSDWAN(sdwanInfo)
+	if len(conf.getDelNodes()) > 0 {
+		t.Errorf("getDelNodes error")
+		return
+	}
+	sdwanInfo2 := SDWANInfo{}
+	sdwanStr = `{"id":1312667996276071700,"name":"network1","gateway":"10.2.3.254/24","mode":"fullmesh","centralNode":"n1-stable","enable":1,"Nodes":[{"name":"222-debug","ip":"10.2.3.13"},{"name":"222stable","ip":"10.2.3.222"},{"name":"5800-debug","ip":"10.2.3.56"},{"name":"Mate60pro","ip":"10.2.3.60"},{"name":"Mymatepad2023","ip":"10.2.3.23"},{"name":"n1-stable","ip":"10.2.3.29","resource":"192.168.3.0/24"},{"name":"tony-stable","ip":"10.2.3.44","resource":"10.1.0.0/16"}]}`
+	if err := json.Unmarshal([]byte(sdwanStr), &sdwanInfo2); err != nil {
+		t.Errorf("unmarshal error")
+		return
+	}
+
+	conf.setSDWAN(sdwanInfo2)
+	diff := conf.getDelNodes()
+	if len(diff) != 1 && diff[0].IP != "10.2.3.4" {
+		t.Errorf("getDelNodes error")
+		return
+	}
+	diff = conf.getAddNodes()
+	if len(diff) != 1 || diff[0].IP != "10.2.3.44" {
+		t.Errorf("getAddNodes error")
+		return
+	}
+}
+func TestSetSDWAN_ClearAll(t *testing.T) {
+	conf := Config{}
+	sdwanInfo := SDWANInfo{}
+	sdwanStr := `{"id":1312667996276071700,"name":"network1","gateway":"10.2.3.254/24","mode":"fullmesh","centralNode":"n1-stable","enable":1,"Nodes":[{"name":"222-debug","ip":"10.2.3.13"},{"name":"222stable","ip":"10.2.3.222"},{"name":"5800-debug","ip":"10.2.3.56"},{"name":"Mate60pro","ip":"10.2.3.60"},{"name":"Mymatepad2023","ip":"10.2.3.23"},{"name":"n1-stable","ip":"10.2.3.29","resource":"192.168.3.0/24"},{"name":"tony-stable","ip":"10.2.3.4","resource":"10.1.0.0/16"}]}`
+	if err := json.Unmarshal([]byte(sdwanStr), &sdwanInfo); err != nil {
+		t.Errorf("unmarshal error")
+		return
+	}
+
+	conf.setSDWAN(sdwanInfo)
+	if len(conf.getDelNodes()) > 0 {
+		t.Errorf("getDelNodes error")
+		return
+	}
+	sdwanInfo2 := SDWANInfo{}
+	sdwanStr = `{"Nodes":null}`
+	if err := json.Unmarshal([]byte(sdwanStr), &sdwanInfo2); err != nil {
+		t.Errorf("unmarshal error")
+		return
+	}
+
+	conf.setSDWAN(sdwanInfo2)
+	diff := conf.getDelNodes()
+	if len(diff) != 7 {
+		t.Errorf("getDelNodes error")
+		return
+	}
+	diff = conf.getAddNodes()
+	if len(diff) != 0 {
+		t.Errorf("getAddNodes error")
+		return
+	}
+}
+func TestSetSDWAN_ChangeNodeResource(t *testing.T) {
+	conf := Config{}
+	sdwanInfo := SDWANInfo{}
+	sdwanStr := `{"id":1312667996276071700,"name":"network1","gateway":"10.2.3.254/24","mode":"fullmesh","centralNode":"n1-stable","enable":1,"Nodes":[{"name":"222-debug","ip":"10.2.3.13"},{"name":"222stable","ip":"10.2.3.222"},{"name":"5800-debug","ip":"10.2.3.56"},{"name":"Mate60pro","ip":"10.2.3.60"},{"name":"Mymatepad2023","ip":"10.2.3.23"},{"name":"n1-stable","ip":"10.2.3.29","resource":"192.168.3.0/24"},{"name":"tony-stable","ip":"10.2.3.4","resource":"10.1.0.0/16"}]}`
+	if err := json.Unmarshal([]byte(sdwanStr), &sdwanInfo); err != nil {
+		t.Errorf("unmarshal error")
+		return
+	}
+
+	conf.setSDWAN(sdwanInfo)
+	if len(conf.getDelNodes()) > 0 {
+		t.Errorf("getDelNodes error")
+		return
+	}
+	sdwanInfo2 := SDWANInfo{}
+	sdwanStr = `{"id":1312667996276071700,"name":"network1","gateway":"10.2.3.254/24","mode":"fullmesh","centralNode":"n1-stable","enable":1,"Nodes":[{"name":"222-debug","ip":"10.2.3.13"},{"name":"222stable","ip":"10.2.3.222"},{"name":"5800-debug","ip":"10.2.3.56"},{"name":"Mate60pro","ip":"10.2.3.60"},{"name":"Mymatepad2023","ip":"10.2.3.23"},{"name":"n1-stable","ip":"10.2.3.29","resource":"192.168.3.0/24"},{"name":"tony-stable","ip":"10.2.3.4","resource":"10.11.0.0/16"}]}`
+	if err := json.Unmarshal([]byte(sdwanStr), &sdwanInfo2); err != nil {
+		t.Errorf("unmarshal error")
+		return
+	}
+
+	conf.setSDWAN(sdwanInfo2)
+	diff := conf.getDelNodes()
+	if len(diff) != 1 && diff[0].IP != "10.2.3.4" {
+		t.Errorf("getDelNodes error")
+		return
+	}
+	diff = conf.getAddNodes()
+	if len(diff) != 1 || diff[0].Resource != "10.11.0.0/16" {
+		t.Errorf("getAddNodes error")
+		return
+	}
+}
+
+func TestInetAtoN(t *testing.T) {
+	ipa, _ := inetAtoN("121.5.147.4")
+	t.Log(ipa)
+	ipa, _ = inetAtoN("121.5.147.4/32")
+	t.Log(ipa)
+}

core/daemon.go

@@ -1,115 +1,115 @@
-package openp2p
+package openp2p
-
+
-import (
+import (
-	"fmt"
+	"fmt"
-	"os"
+	"os"
-	"path/filepath"
+	"path/filepath"
-	"time"
+	"time"
-
+
-	"github.com/openp2p-cn/service"
+	"github.com/openp2p-cn/service"
-)
+)
-
+
-type daemon struct {
+type daemon struct {
-	running bool
+	running bool
-	proc    *os.Process
+	proc    *os.Process
-}
+}
-
+
-func (d *daemon) Start(s service.Service) error {
+func (d *daemon) Start(s service.Service) error {
-	gLog.Println(LvINFO, "daemon start")
+	gLog.Println(LvINFO, "daemon start")
-	return nil
+	return nil
-}
+}
-
+
-func (d *daemon) Stop(s service.Service) error {
+func (d *daemon) Stop(s service.Service) error {
-	gLog.Println(LvINFO, "service stop")
+	gLog.Println(LvINFO, "service stop")
-	d.running = false
+	d.running = false
-	if d.proc != nil {
+	if d.proc != nil {
-		gLog.Println(LvINFO, "stop worker")
+		gLog.Println(LvINFO, "stop worker")
-		d.proc.Kill()
+		d.proc.Kill()
-	}
+	}
-	if service.Interactive() {
+	if service.Interactive() {
-		gLog.Println(LvINFO, "stop daemon")
+		gLog.Println(LvINFO, "stop daemon")
-		os.Exit(0)
+		os.Exit(0)
-	}
+	}
-	return nil
+	return nil
-}
+}
-
+
-func (d *daemon) run() {
+func (d *daemon) run() {
-	gLog.Println(LvINFO, "daemon run start")
+	gLog.Println(LvINFO, "daemon run start")
-	defer gLog.Println(LvINFO, "daemon run end")
+	defer gLog.Println(LvINFO, "daemon run end")
-	d.running = true
+	d.running = true
-	binPath, _ := os.Executable()
+	binPath, _ := os.Executable()
-	mydir, err := os.Getwd()
+	mydir, err := os.Getwd()
-	if err != nil {
+	if err != nil {
-		fmt.Println(err)
+		fmt.Println(err)
-	}
+	}
-	gLog.Println(LvINFO, mydir)
+	gLog.Println(LvINFO, mydir)
-	conf := &service.Config{
+	conf := &service.Config{
-		Name:        ProductName,
+		Name:        ProductName,
-		DisplayName: ProductName,
+		DisplayName: ProductName,
-		Description: ProductName,
+		Description: ProductName,
-		Executable:  binPath,
+		Executable:  binPath,
-	}
+	}
-
+
-	s, _ := service.New(d, conf)
+	s, _ := service.New(d, conf)
-	go s.Run()
+	go s.Run()
-	var args []string
+	var args []string
-	// rm -d parameter
+	// rm -d parameter
-	for i := 0; i < len(os.Args); i++ {
+	for i := 0; i < len(os.Args); i++ {
-		if os.Args[i] == "-d" {
+		if os.Args[i] == "-d" {
-			args = append(os.Args[0:i], os.Args[i+1:]...)
+			args = append(os.Args[0:i], os.Args[i+1:]...)
-			break
+			break
-		}
+		}
-	}
+	}
-	args = append(args, "-nv")
+	args = append(args, "-nv")
-	for {
+	for {
-		// start worker
+		// start worker
-		tmpDump := filepath.Join("log", "dump.log.tmp")
+		tmpDump := filepath.Join("log", "dump.log.tmp")
-		dumpFile := filepath.Join("log", "dump.log")
+		dumpFile := filepath.Join("log", "dump.log")
-		f, err := os.Create(filepath.Join(tmpDump))
+		f, err := os.Create(filepath.Join(tmpDump))
-		if err != nil {
+		if err != nil {
-			gLog.Printf(LvERROR, "start worker error:%s", err)
+			gLog.Printf(LvERROR, "start worker error:%s", err)
-			return
+			return
-		}
+		}
-		gLog.Println(LvINFO, "start worker process, args:", args)
+		gLog.Println(LvINFO, "start worker process, args:", args)
-		execSpec := &os.ProcAttr{Env: append(os.Environ(), "GOTRACEBACK=crash"), Files: []*os.File{os.Stdin, os.Stdout, f}}
+		execSpec := &os.ProcAttr{Env: append(os.Environ(), "GOTRACEBACK=crash"), Files: []*os.File{os.Stdin, os.Stdout, f}}
-		p, err := os.StartProcess(binPath, args, execSpec)
+		p, err := os.StartProcess(binPath, args, execSpec)
-		if err != nil {
+		if err != nil {
-			gLog.Printf(LvERROR, "start worker error:%s", err)
+			gLog.Printf(LvERROR, "start worker error:%s", err)
-			return
+			return
-		}
+		}
-		d.proc = p
+		d.proc = p
-		_, _ = p.Wait()
+		_, _ = p.Wait()
-		f.Close()
+		f.Close()
-		time.Sleep(time.Second)
+		time.Sleep(time.Second)
-		err = os.Rename(tmpDump, dumpFile)
+		err = os.Rename(tmpDump, dumpFile)
-		if err != nil {
+		if err != nil {
-			gLog.Printf(LvERROR, "rename dump error:%s", err)
+			gLog.Printf(LvERROR, "rename dump error:%s", err)
-		}
+		}
-		if !d.running {
+		if !d.running {
-			return
+			return
-		}
+		}
-		gLog.Printf(LvERROR, "worker stop, restart it after 10s")
+		gLog.Printf(LvERROR, "worker stop, restart it after 10s")
-		time.Sleep(time.Second * 10)
+		time.Sleep(time.Second * 10)
-	}
+	}
-}
+}
-
+
-func (d *daemon) Control(ctrlComm string, exeAbsPath string, args []string) error {
+func (d *daemon) Control(ctrlComm string, exeAbsPath string, args []string) error {
-	svcConfig := &service.Config{
+	svcConfig := &service.Config{
-		Name:        ProductName,
+		Name:        ProductName,
-		DisplayName: ProductName,
+		DisplayName: ProductName,
-		Description: ProductName,
+		Description: ProductName,
-		Executable:  exeAbsPath,
+		Executable:  exeAbsPath,
-		Arguments:   args,
+		Arguments:   args,
-	}
+	}
-
+
-	s, e := service.New(d, svcConfig)
+	s, e := service.New(d, svcConfig)
-	if e != nil {
+	if e != nil {
-		return e
+		return e
-	}
+	}
-	e = service.Control(s, ctrlComm)
+	e = service.Control(s, ctrlComm)
-	if e != nil {
+	if e != nil {
-		return e
+		return e
-	}
+	}
-
+
-	return nil
+	return nil
-}
+}

core/errorcode.go

@@ -8,12 +8,26 @@ import (
 var (
 	// ErrorS2S string = "s2s is not supported"
 	// ErrorHandshake string = "handshake error"
-	ErrorS2S                = errors.New("s2s is not supported")
+	ErrorS2S                 = errors.New("s2s is not supported")
-	ErrorHandshake          = errors.New("handshake error")
+	ErrorHandshake           = errors.New("handshake error")
-	ErrorNewUser            = errors.New("new user")
+	ErrorNewUser             = errors.New("new user")
-	ErrorLogin              = errors.New("user or password not correct")
+	ErrorLogin               = errors.New("user or password not correct")
-	ErrNodeTooShort         = errors.New("node name too short, it must >=8 charaters")
+	ErrNodeTooShort          = errors.New("node name too short, it must >=8 charaters")
-	ErrPeerOffline          = errors.New("peer offline")
+	ErrReadDB                = errors.New("read db error")
-	ErrMsgFormat            = errors.New("message format wrong")
+	ErrNoUpdate              = errors.New("there are currently no updates available")
-	ErrVersionNotCompatible = errors.New("version not compatible")
+	ErrPeerOffline           = errors.New("peer offline")
+	ErrNetwork               = errors.New("network error")
+	ErrMsgFormat             = errors.New("message format wrong")
+	ErrVersionNotCompatible  = errors.New("version not compatible")
+	ErrOverlayConnDisconnect = errors.New("overlay connection is disconnected")
+	ErrConnectRelayNode      = errors.New("connect relay node error")
+	ErrConnectPublicV4       = errors.New("connect public ipv4 error")
+	ErrMsgChannelNotFound    = errors.New("message channel not found")
+	ErrRelayTunnelNotFound   = errors.New("relay tunnel not found")
+	ErrSymmetricLimit        = errors.New("symmetric limit")
+	ErrForceRelay            = errors.New("force relay")
+	ErrPeerConnectRelay      = errors.New("peer connect relayNode error")
+	ErrBuildTunnelBusy       = errors.New("build tunnel busy")
+	ErrMemAppTunnelNotFound  = errors.New("memapp tunnel not found")
+	ErrRemoteServiceUnable   = errors.New("remote service unable")
 )

core/handlepush.go

@@ -1,281 +1,482 @@
-package openp2p
+package openp2p
-
+
-import (
+import (
-	"bytes"
+	"bytes"
-	"encoding/binary"
+	"encoding/binary"
-	"encoding/json"
+	"encoding/json"
-	"fmt"
+	"fmt"
-	"os"
+	"net"
-	"os/exec"
+	"os"
-	"path/filepath"
+	"path/filepath"
-	"time"
+	"reflect"
-)
+	"runtime"
-
+	"time"
-func handlePush(pn *P2PNetwork, subType uint16, msg []byte) error {
+
-	pushHead := PushHeader{}
+	"github.com/openp2p-cn/totp"
-	err := binary.Read(bytes.NewReader(msg[openP2PHeaderSize:openP2PHeaderSize+PushHeaderSize]), binary.LittleEndian, &pushHead)
+)
-	if err != nil {
+
-		return err
+func handlePush(subType uint16, msg []byte) error {
-	}
+	pushHead := PushHeader{}
-	gLog.Printf(LvDEBUG, "handle push msg type:%d, push header:%+v", subType, pushHead)
+	err := binary.Read(bytes.NewReader(msg[openP2PHeaderSize:openP2PHeaderSize+PushHeaderSize]), binary.LittleEndian, &pushHead)
-	switch subType {
+	if err != nil {
-	case MsgPushConnectReq: // TODO: handle a msg move to a new function
+		return err
-		req := PushConnectReq{}
+	}
-		err := json.Unmarshal(msg[openP2PHeaderSize+PushHeaderSize:], &req)
+	gLog.Printf(LvDEBUG, "handle push msg type:%d, push header:%+v", subType, pushHead)
-		if err != nil {
+	switch subType {
-			gLog.Printf(LvERROR, "wrong MsgPushConnectReq:%s", err)
+	case MsgPushConnectReq:
-			return err
+		err = handleConnectReq(msg)
-		}
+	case MsgPushRsp:
-		gLog.Printf(LvINFO, "%s is connecting...", req.From)
+		rsp := PushRsp{}
-		gLog.Println(LvDEBUG, "push connect response to ", req.From)
+		if err = json.Unmarshal(msg[openP2PHeaderSize:], &rsp); err != nil {
-		if compareVersion(req.Version, LeastSupportVersion) == LESS {
+			gLog.Printf(LvERROR, "wrong pushRsp:%s", err)
-			gLog.Println(LvERROR, ErrVersionNotCompatible.Error(), ":", req.From)
+			return err
-			rsp := PushConnectRsp{
+		}
-				Error:  10,
+		if rsp.Error == 0 {
-				Detail: ErrVersionNotCompatible.Error(),
+			gLog.Printf(LvDEBUG, "push ok, detail:%s", rsp.Detail)
-				To:     req.From,
+		} else {
-				From:   pn.config.Node,
+			gLog.Printf(LvERROR, "push error:%d, detail:%s", rsp.Error, rsp.Detail)
-			}
+		}
-			pn.push(req.From, MsgPushConnectRsp, rsp)
+	case MsgPushAddRelayTunnelReq:
-			return ErrVersionNotCompatible
+		req := AddRelayTunnelReq{}
-		}
+		if err = json.Unmarshal(msg[openP2PHeaderSize+PushHeaderSize:], &req); err != nil {
-		// verify totp token or token
+			gLog.Printf(LvERROR, "wrong %v:%s", reflect.TypeOf(req), err)
-		if VerifyTOTP(req.Token, pn.config.Token, time.Now().Unix()+(pn.serverTs-pn.localTs)) || // localTs may behind, auto adjust ts
+			return err
-			VerifyTOTP(req.Token, pn.config.Token, time.Now().Unix()) {
+		}
-			gLog.Printf(LvINFO, "Access Granted\n")
+		config := AppConfig{}
-			config := AppConfig{}
+		config.PeerNode = req.RelayName
-			config.peerNatType = req.NatType
+		config.peerToken = req.RelayToken
-			config.peerConeNatPort = req.ConeNatPort
+		config.relayMode = req.RelayMode
-			config.peerIP = req.FromIP
+		go func(r AddRelayTunnelReq) {
-			config.PeerNode = req.From
+			t, errDt := GNetwork.addDirectTunnel(config, 0)
-			config.peerVersion = req.Version
+			if errDt == nil {
-			config.fromToken = req.Token
+				// notify peer relay ready
-			config.peerIPv6 = req.IPv6
+				msg := TunnelMsg{ID: t.id}
-			config.hasIPv4 = req.HasIPv4
+				GNetwork.push(r.From, MsgPushAddRelayTunnelRsp, msg)
-			config.hasUPNPorNATPMP = req.HasUPNPorNATPMP
+				appConfig := config
-			config.linkMode = req.LinkMode
+				appConfig.PeerNode = req.From
-			config.isUnderlayServer = req.IsUnderlayServer
+			} else {
-			// share relay node will limit bandwidth
+				gLog.Printf(LvERROR, "addDirectTunnel error:%s", errDt)
-			if req.Token != pn.config.Token {
+				GNetwork.push(r.From, MsgPushAddRelayTunnelRsp, "error") // compatible with old version client, trigger unmarshal error
-				gLog.Printf(LvINFO, "set share bandwidth %d mbps", pn.config.ShareBandwidth)
+			}
-				config.shareBandwidth = pn.config.ShareBandwidth
+		}(req)
-			}
+	case MsgPushServerSideSaveMemApp:
-			// go pn.AddTunnel(config, req.ID)
+		req := ServerSideSaveMemApp{}
-			go pn.addDirectTunnel(config, req.ID)
+		if err = json.Unmarshal(msg[openP2PHeaderSize+PushHeaderSize:], &req); err != nil {
-			break
+			gLog.Printf(LvERROR, "wrong %v:%s", reflect.TypeOf(req), err)
-		}
+			return err
-		gLog.Println(LvERROR, "Access Denied:", req.From)
+		}
-		rsp := PushConnectRsp{
+		gLog.Println(LvDEBUG, "handle MsgPushServerSideSaveMemApp:", prettyJson(req))
-			Error:  1,
+		var existTunnel *P2PTunnel
-			Detail: fmt.Sprintf("connect to %s error: Access Denied", pn.config.Node),
+		i, ok := GNetwork.allTunnels.Load(req.TunnelID)
-			To:     req.From,
+		if !ok {
-			From:   pn.config.Node,
+			time.Sleep(time.Millisecond * 100)
-		}
+			i, ok = GNetwork.allTunnels.Load(req.TunnelID) // retry sometimes will receive MsgPushServerSideSaveMemApp but p2ptunnel not store yet.
-		pn.push(req.From, MsgPushConnectRsp, rsp)
+			if !ok {
-	case MsgPushRsp:
+				gLog.Println(LvERROR, "handle MsgPushServerSideSaveMemApp error:", ErrMemAppTunnelNotFound)
-		rsp := PushRsp{}
+				return ErrMemAppTunnelNotFound
-		err := json.Unmarshal(msg[openP2PHeaderSize:], &rsp)
+			}
-		if err != nil {
+		}
-			gLog.Printf(LvERROR, "wrong pushRsp:%s", err)
+		existTunnel = i.(*P2PTunnel)
-			return err
+		peerID := NodeNameToID(req.From)
-		}
+		existApp, appok := GNetwork.apps.Load(peerID)
-		if rsp.Error == 0 {
+		if appok {
-			gLog.Printf(LvDEBUG, "push ok, detail:%s", rsp.Detail)
+			app := existApp.(*p2pApp)
-		} else {
+			app.config.AppName = fmt.Sprintf("%d", peerID)
-			gLog.Printf(LvERROR, "push error:%d, detail:%s", rsp.Error, rsp.Detail)
+			app.id = req.AppID
-		}
+			app.setRelayTunnelID(req.RelayTunnelID)
-	case MsgPushAddRelayTunnelReq:
+			app.relayMode = req.RelayMode
-		req := AddRelayTunnelReq{}
+			app.hbTimeRelay = time.Now()
-		err := json.Unmarshal(msg[openP2PHeaderSize+PushHeaderSize:], &req)
+			if req.RelayTunnelID == 0 {
-		if err != nil {
+				app.setDirectTunnel(existTunnel)
-			gLog.Printf(LvERROR, "wrong RelayNodeRsp:%s", err)
+			} else {
-			return err
+				app.setRelayTunnel(existTunnel)
-		}
+			}
-		config := AppConfig{}
+			gLog.Println(LvDEBUG, "find existing memapp, update it")
-		config.PeerNode = req.RelayName
+		} else {
-		config.peerToken = req.RelayToken
+			appConfig := existTunnel.config
-		go func(r AddRelayTunnelReq) {
+			appConfig.SrcPort = 0
-			t, errDt := pn.addDirectTunnel(config, 0)
+			appConfig.Protocol = ""
-			if errDt == nil {
+			appConfig.AppName = fmt.Sprintf("%d", peerID)
-				// notify peer relay ready
+			appConfig.PeerNode = req.From
-				msg := TunnelMsg{ID: t.id}
+			app := p2pApp{
-				pn.push(r.From, MsgPushAddRelayTunnelRsp, msg)
+				id:          req.AppID,
-			}
+				config:      appConfig,
-		}(req)
+				relayMode:   req.RelayMode,
-	case MsgPushAPPKey:
+				running:     true,
-		req := APPKeySync{}
+				hbTimeRelay: time.Now(),
-		err := json.Unmarshal(msg[openP2PHeaderSize+PushHeaderSize:], &req)
+			}
-		if err != nil {
+			if req.RelayTunnelID == 0 {
-			gLog.Printf(LvERROR, "wrong APPKeySync:%s", err)
+				app.setDirectTunnel(existTunnel)
-			return err
+			} else {
-		}
+				app.setRelayTunnel(existTunnel)
-		SaveKey(req.AppID, req.AppKey)
+				app.setRelayTunnelID(req.RelayTunnelID)
-	case MsgPushUpdate:
+			}
-		gLog.Println(LvINFO, "MsgPushUpdate")
+			if req.RelayTunnelID != 0 {
-		update(pn.config.ServerHost, pn.config.ServerPort) // download new version first, then exec ./openp2p update
+				app.relayNode = req.Node
-		targetPath := filepath.Join(defaultInstallPath, defaultBinName)
+			}
-		args := []string{"update"}
+			GNetwork.apps.Store(NodeNameToID(req.From), &app)
-		env := os.Environ()
+		}
-		cmd := exec.Command(targetPath, args...)
+
-		cmd.Stdout = os.Stdout
+		return nil
-		cmd.Stderr = os.Stderr
+	case MsgPushAPPKey:
-		cmd.Stdin = os.Stdin
+		req := APPKeySync{}
-		cmd.Env = env
+		if err = json.Unmarshal(msg[openP2PHeaderSize+PushHeaderSize:], &req); err != nil {
-		err := cmd.Run()
+			gLog.Printf(LvERROR, "wrong %v:%s", reflect.TypeOf(req), err)
-		if err == nil {
+			return err
-			os.Exit(0)
+		}
-		}
+		SaveKey(req.AppID, req.AppKey)
-		return err
+	case MsgPushUpdate:
-	case MsgPushRestart:
+		gLog.Println(LvINFO, "MsgPushUpdate")
-		gLog.Println(LvINFO, "MsgPushRestart")
+		err := update(gConf.Network.ServerHost, gConf.Network.ServerPort)
-		os.Exit(0)
+		if err == nil {
-		return err
+			os.Exit(0)
-	case MsgPushReportApps:
+		}
-		gLog.Println(LvINFO, "MsgPushReportApps")
+		return err
-		req := ReportApps{}
+	case MsgPushRestart:
-		gConf.mtx.Lock()
+		gLog.Println(LvINFO, "MsgPushRestart")
-		defer gConf.mtx.Unlock()
+		os.Exit(0)
-		for _, config := range gConf.Apps {
+		return err
-			appActive := 0
+	case MsgPushReportApps:
-			relayNode := ""
+		err = handleReportApps()
-			relayMode := ""
+	case MsgPushReportMemApps:
-			linkMode := LinkModeUDPPunch
+		err = handleReportMemApps()
-			i, ok := pn.apps.Load(fmt.Sprintf("%s%d", config.Protocol, config.SrcPort))
+	case MsgPushReportLog:
-			if ok {
+		err = handleLog(msg)
-				app := i.(*p2pApp)
+	case MsgPushReportGoroutine:
-				if app.isActive() {
+		err = handleReportGoroutine()
-					appActive = 1
+	case MsgPushCheckRemoteService:
-				}
+		err = handleCheckRemoteService(msg)
-				relayNode = app.relayNode
+	case MsgPushEditApp:
-				relayMode = app.relayMode
+		err = handleEditApp(msg)
-				linkMode = app.tunnel.linkModeWeb
+	case MsgPushEditNode:
-			}
+		gLog.Println(LvINFO, "MsgPushEditNode")
-			appInfo := AppInfo{
+		req := EditNode{}
-				AppName:     config.AppName,
+		if err = json.Unmarshal(msg[openP2PHeaderSize:], &req); err != nil {
-				Error:       config.errMsg,
+			gLog.Printf(LvERROR, "wrong %v:%s  %s", reflect.TypeOf(req), err, string(msg[openP2PHeaderSize:]))
-				Protocol:    config.Protocol,
+			return err
-				SrcPort:     config.SrcPort,
+		}
-				RelayNode:   relayNode,
+		gConf.setNode(req.NewName)
-				RelayMode:   relayMode,
+		gConf.setShareBandwidth(req.Bandwidth)
-				LinkMode:    linkMode,
+		os.Exit(0)
-				PeerNode:    config.PeerNode,
+	case MsgPushSwitchApp:
-				DstHost:     config.DstHost,
+		gLog.Println(LvINFO, "MsgPushSwitchApp")
-				DstPort:     config.DstPort,
+		app := AppInfo{}
-				PeerUser:    config.PeerUser,
+		if err = json.Unmarshal(msg[openP2PHeaderSize:], &app); err != nil {
-				PeerIP:      config.peerIP,
+			gLog.Printf(LvERROR, "wrong %v:%s  %s", reflect.TypeOf(app), err, string(msg[openP2PHeaderSize:]))
-				PeerNatType: config.peerNatType,
+			return err
-				RetryTime:   config.retryTime.Local().Format("2006-01-02T15:04:05-0700"),
+		}
-				ConnectTime: config.connectTime.Local().Format("2006-01-02T15:04:05-0700"),
+		config := AppConfig{Enabled: app.Enabled, SrcPort: app.SrcPort, Protocol: app.Protocol}
-				IsActive:    appActive,
+		gLog.Println(LvINFO, app.AppName, " switch to ", app.Enabled)
-				Enabled:     config.Enabled,
+		gConf.switchApp(config, app.Enabled)
-			}
+		if app.Enabled == 0 {
-			req.Apps = append(req.Apps, appInfo)
+			// disable APP
-		}
+			GNetwork.DeleteApp(config)
-		pn.write(MsgReport, MsgReportApps, &req)
+		}
-	case MsgPushReportLog:
+	case MsgPushDstNodeOnline:
-		gLog.Println(LvINFO, "MsgPushReportLog")
+		gLog.Println(LvINFO, "MsgPushDstNodeOnline")
-		req := ReportLogReq{}
+		req := PushDstNodeOnline{}
-		err := json.Unmarshal(msg[openP2PHeaderSize:], &req)
+		if err = json.Unmarshal(msg[openP2PHeaderSize:], &req); err != nil {
-		if err != nil {
+			gLog.Printf(LvERROR, "wrong %v:%s  %s", reflect.TypeOf(req), err, string(msg[openP2PHeaderSize:]))
-			gLog.Printf(LvERROR, "wrong MsgPushReportLog:%s  %s", err, string(msg[openP2PHeaderSize:]))
+			return err
-			return err
+		}
-		}
+		gLog.Println(LvINFO, "retry peerNode ", req.Node)
-		if req.FileName == "" {
+		gConf.retryApp(req.Node)
-			req.FileName = "openp2p.log"
+	default:
-		}
+		i, ok := GNetwork.msgMap.Load(pushHead.From)
-		f, err := os.Open(filepath.Join("log", req.FileName))
+		if !ok {
-		if err != nil {
+			return ErrMsgChannelNotFound
-			gLog.Println(LvERROR, "read log file error:", err)
+		}
-			break
+		ch := i.(chan msgCtx)
-		}
+		ch <- msgCtx{data: msg, ts: time.Now()}
-		fi, err := f.Stat()
+	}
-		if err != nil {
+	return err
-			break
+}
-		}
+
-		if req.Offset == 0 && fi.Size() > 4096 {
+func handleEditApp(msg []byte) (err error) {
-			req.Offset = fi.Size() - 4096
+	gLog.Println(LvINFO, "MsgPushEditApp")
-		}
+	newApp := AppInfo{}
-		if req.Len <= 0 {
+	if err = json.Unmarshal(msg[openP2PHeaderSize:], &newApp); err != nil {
-			req.Len = 4096
+		gLog.Printf(LvERROR, "wrong %v:%s  %s", reflect.TypeOf(newApp), err, string(msg[openP2PHeaderSize:]))
-		}
+		return err
-		f.Seek(req.Offset, 0)
+	}
-		if req.Len > 1024*1024 { // too large
+	oldConf := AppConfig{Enabled: 1}
-			break
+	// protocol0+srcPort0 exist, delApp
-		}
+	oldConf.AppName = newApp.AppName
-		buff := make([]byte, req.Len)
+	oldConf.Protocol = newApp.Protocol0
-		readLength, err := f.Read(buff)
+	oldConf.Whitelist = newApp.Whitelist
-		f.Close()
+	oldConf.SrcPort = newApp.SrcPort0
-		if err != nil {
+	oldConf.PeerNode = newApp.PeerNode
-			gLog.Println(LvERROR, "read log content error:", err)
+	oldConf.DstHost = newApp.DstHost
-			break
+	oldConf.DstPort = newApp.DstPort
-		}
+	if newApp.Protocol0 != "" && newApp.SrcPort0 != 0 { // not edit
-		rsp := ReportLogRsp{}
+		gConf.delete(oldConf)
-		rsp.Content = string(buff[:readLength])
+	}
-		rsp.FileName = req.FileName
+
-		rsp.Total = fi.Size()
+	// AddApp
-		rsp.Len = req.Len
+	newConf := oldConf
-		pn.write(MsgReport, MsgPushReportLog, &rsp)
+	newConf.Protocol = newApp.Protocol
-	case MsgPushEditApp:
+	newConf.SrcPort = newApp.SrcPort
-		gLog.Println(LvINFO, "MsgPushEditApp")
+	newConf.RelayNode = newApp.SpecRelayNode
-		newApp := AppInfo{}
+	newConf.PunchPriority = newApp.PunchPriority
-		err := json.Unmarshal(msg[openP2PHeaderSize:], &newApp)
+	gConf.add(newConf, false)
-		if err != nil {
+	if newApp.Protocol0 != "" && newApp.SrcPort0 != 0 { // not edit
-			gLog.Printf(LvERROR, "wrong MsgPushEditApp:%s  %s", err, string(msg[openP2PHeaderSize:]))
+		GNetwork.DeleteApp(oldConf) // DeleteApp may cost some times, execute at the end
-			return err
+	}
-		}
+	return nil
-		oldConf := AppConfig{Enabled: 1}
+}
-		// protocol0+srcPort0 exist, delApp
+
-		oldConf.AppName = newApp.AppName
+func handleConnectReq(msg []byte) (err error) {
-		oldConf.Protocol = newApp.Protocol0
+	req := PushConnectReq{}
-		oldConf.SrcPort = newApp.SrcPort0
+	if err = json.Unmarshal(msg[openP2PHeaderSize+PushHeaderSize:], &req); err != nil {
-		oldConf.PeerNode = newApp.PeerNode
+		gLog.Printf(LvERROR, "wrong %v:%s", reflect.TypeOf(req), err)
-		oldConf.DstHost = newApp.DstHost
+		return err
-		oldConf.DstPort = newApp.DstPort
+	}
-
+	gLog.Printf(LvDEBUG, "%s is connecting...", req.From)
-		gConf.delete(oldConf)
+	gLog.Println(LvDEBUG, "push connect response to ", req.From)
-		// AddApp
+	if compareVersion(req.Version, LeastSupportVersion) < 0 {
-		newConf := oldConf
+		gLog.Println(LvERROR, ErrVersionNotCompatible.Error(), ":", req.From)
-		newConf.Protocol = newApp.Protocol
+		rsp := PushConnectRsp{
-		newConf.SrcPort = newApp.SrcPort
+			Error:  10,
-		gConf.add(newConf, false)
+			Detail: ErrVersionNotCompatible.Error(),
-		gConf.save()          // save quickly for the next request reportApplist
+			To:     req.From,
-		pn.DeleteApp(oldConf) // DeleteApp may cost some times, execute at the end
+			From:   gConf.Network.Node,
-		// autoReconnect will auto AddApp
+		}
-		// pn.AddApp(config)
+		GNetwork.push(req.From, MsgPushConnectRsp, rsp)
-		// TODO: report result
+		return ErrVersionNotCompatible
-	case MsgPushEditNode:
+	}
-		gLog.Println(LvINFO, "MsgPushEditNode")
+	// verify totp token or token
-		req := EditNode{}
+	t := totp.TOTP{Step: totp.RelayTOTPStep}
-		err := json.Unmarshal(msg[openP2PHeaderSize:], &req)
+	if t.Verify(req.Token, gConf.Network.Token, time.Now().Unix()-GNetwork.dt/int64(time.Second)) { // localTs may behind, auto adjust ts
-		if err != nil {
+		gLog.Printf(LvINFO, "Access Granted")
-			gLog.Printf(LvERROR, "wrong MsgPushEditNode:%s  %s", err, string(msg[openP2PHeaderSize:]))
+		config := AppConfig{}
-			return err
+		config.peerNatType = req.NatType
-		}
+		config.peerConeNatPort = req.ConeNatPort
-		gConf.setNode(req.NewName)
+		config.peerIP = req.FromIP
-		gConf.setShareBandwidth(req.Bandwidth)
+		config.PeerNode = req.From
-		gConf.save()
+		config.peerVersion = req.Version
-		// TODO: hot reload
+		config.fromToken = req.Token
-		os.Exit(0)
+		config.peerIPv6 = req.IPv6
-	case MsgPushSwitchApp:
+		config.hasIPv4 = req.HasIPv4
-		gLog.Println(LvINFO, "MsgPushSwitchApp")
+		config.hasUPNPorNATPMP = req.HasUPNPorNATPMP
-		app := AppInfo{}
+		config.linkMode = req.LinkMode
-		err := json.Unmarshal(msg[openP2PHeaderSize:], &app)
+		config.isUnderlayServer = req.IsUnderlayServer
-		if err != nil {
+		config.UnderlayProtocol = req.UnderlayProtocol
-			gLog.Printf(LvERROR, "wrong MsgPushSwitchApp:%s  %s", err, string(msg[openP2PHeaderSize:]))
+		// share relay node will limit bandwidth
-			return err
+		if req.Token != gConf.Network.Token {
-		}
+			gLog.Printf(LvINFO, "set share bandwidth %d mbps", gConf.Network.ShareBandwidth)
-		config := AppConfig{Enabled: app.Enabled, SrcPort: app.SrcPort, Protocol: app.Protocol}
+			config.shareBandwidth = gConf.Network.ShareBandwidth
-		gLog.Println(LvINFO, app.AppName, " switch to ", app.Enabled)
+		}
-		gConf.switchApp(config, app.Enabled)
+		// go GNetwork.AddTunnel(config, req.ID)
-		if app.Enabled == 0 {
+		go func() {
-			// disable APP
+			GNetwork.addDirectTunnel(config, req.ID)
-			pn.DeleteApp(config)
+		}()
-		}
+		return nil
-	default:
+	}
-		pn.msgMapMtx.Lock()
+	gLog.Println(LvERROR, "Access Denied:", req.From)
-		ch := pn.msgMap[pushHead.From]
+	rsp := PushConnectRsp{
-		pn.msgMapMtx.Unlock()
+		Error:  1,
-		ch <- msg
+		Detail: fmt.Sprintf("connect to %s error: Access Denied", gConf.Network.Node),
-	}
+		To:     req.From,
-	return nil
+		From:   gConf.Network.Node,
-}
+	}
+	return GNetwork.push(req.From, MsgPushConnectRsp, rsp)
+}
+
+func handleReportApps() (err error) {
+	gLog.Println(LvINFO, "MsgPushReportApps")
+	req := ReportApps{}
+	gConf.mtx.Lock()
+	defer gConf.mtx.Unlock()
+
+	for _, config := range gConf.Apps {
+		appActive := 0
+		relayNode := ""
+		specRelayNode := ""
+		relayMode := ""
+		linkMode := LinkModeUDPPunch
+		var connectTime string
+		var retryTime string
+		var app *p2pApp
+		i, ok := GNetwork.apps.Load(config.ID())
+		if ok {
+			app = i.(*p2pApp)
+			if app.isActive() {
+				appActive = 1
+			}
+			if app.config.SrcPort == 0 { // memapp
+				continue
+			}
+			specRelayNode = app.config.RelayNode
+			if !app.isDirect() { // TODO: should always report relay node for app edit
+				relayNode = app.relayNode
+				relayMode = app.relayMode
+			}
+
+			if app.Tunnel() != nil {
+				linkMode = app.Tunnel().linkModeWeb
+			}
+			retryTime = app.RetryTime().Local().Format("2006-01-02T15:04:05-0700")
+			connectTime = app.ConnectTime().Local().Format("2006-01-02T15:04:05-0700")
+
+		}
+		appInfo := AppInfo{
+			AppName:       config.AppName,
+			Error:         config.errMsg,
+			Protocol:      config.Protocol,
+			PunchPriority: config.PunchPriority,
+			Whitelist:     config.Whitelist,
+			SrcPort:       config.SrcPort,
+			RelayNode:     relayNode,
+			SpecRelayNode: specRelayNode,
+			RelayMode:     relayMode,
+			LinkMode:      linkMode,
+			PeerNode:      config.PeerNode,
+			DstHost:       config.DstHost,
+			DstPort:       config.DstPort,
+			PeerUser:      config.PeerUser,
+			PeerIP:        config.peerIP,
+			PeerNatType:   config.peerNatType,
+			RetryTime:     retryTime,
+			ConnectTime:   connectTime,
+			IsActive:      appActive,
+			Enabled:       config.Enabled,
+		}
+		req.Apps = append(req.Apps, appInfo)
+	}
+	return GNetwork.write(MsgReport, MsgReportApps, &req)
+
+}
+
+func handleReportMemApps() (err error) {
+	gLog.Println(LvINFO, "handleReportMemApps")
+	req := ReportApps{}
+	gConf.mtx.Lock()
+	defer gConf.mtx.Unlock()
+	GNetwork.sdwan.sysRoute.Range(func(key, value interface{}) bool {
+		node := value.(*sdwanNode)
+		appActive := 0
+		relayMode := ""
+		var connectTime string
+		var retryTime string
+
+		i, ok := GNetwork.apps.Load(node.id)
+		var app *p2pApp
+		if ok {
+			app = i.(*p2pApp)
+			if app.isActive() {
+				appActive = 1
+			}
+			if !app.isDirect() {
+				relayMode = app.relayMode
+			}
+			retryTime = app.RetryTime().Local().Format("2006-01-02T15:04:05-0700")
+			connectTime = app.ConnectTime().Local().Format("2006-01-02T15:04:05-0700")
+		}
+		appInfo := AppInfo{
+			RelayMode: relayMode,
+			PeerNode:  node.name,
+			IsActive:  appActive,
+			Enabled:   1,
+		}
+		if app != nil {
+			appInfo.AppName = app.config.AppName
+			appInfo.Error = app.config.errMsg
+			appInfo.Protocol = app.config.Protocol
+			appInfo.Whitelist = app.config.Whitelist
+			appInfo.SrcPort = app.config.SrcPort
+			if !app.isDirect() {
+				appInfo.RelayNode = app.relayNode
+			}
+
+			if app.Tunnel() != nil {
+				appInfo.LinkMode = app.Tunnel().linkModeWeb
+			}
+			appInfo.DstHost = app.config.DstHost
+			appInfo.DstPort = app.config.DstPort
+			appInfo.PeerUser = app.config.PeerUser
+			appInfo.PeerIP = app.config.peerIP
+			appInfo.PeerNatType = app.config.peerNatType
+			appInfo.RetryTime = retryTime
+			appInfo.ConnectTime = connectTime
+		}
+		req.Apps = append(req.Apps, appInfo)
+		return true
+	})
+	gLog.Println(LvDEBUG, "handleReportMemApps res:", prettyJson(req))
+	return GNetwork.write(MsgReport, MsgReportMemApps, &req)
+}
+
+func handleLog(msg []byte) (err error) {
+	gLog.Println(LvDEBUG, "MsgPushReportLog")
+	const defaultLen = 1024 * 128
+	const maxLen = 1024 * 1024
+	req := ReportLogReq{}
+	if err = json.Unmarshal(msg[openP2PHeaderSize:], &req); err != nil {
+		gLog.Printf(LvERROR, "wrong %v:%s  %s", reflect.TypeOf(req), err, string(msg[openP2PHeaderSize:]))
+		return err
+	}
+	if req.FileName == "" {
+		req.FileName = "openp2p.log"
+	} else {
+		req.FileName = sanitizeFileName(req.FileName)
+	}
+	f, err := os.Open(filepath.Join("log", req.FileName))
+	if err != nil {
+		gLog.Println(LvERROR, "read log file error:", err)
+		return err
+	}
+	fi, err := f.Stat()
+	if err != nil {
+		return err
+	}
+	if req.Offset > fi.Size() {
+		req.Offset = fi.Size() - defaultLen
+	}
+	// verify input parameters
+	if req.Offset < 0 {
+		req.Offset = 0
+	}
+	if req.Len <= 0 || req.Len > maxLen {
+		req.Len = defaultLen
+	}
+
+	f.Seek(req.Offset, 0)
+	buff := make([]byte, req.Len)
+	readLength, err := f.Read(buff)
+	f.Close()
+	if err != nil {
+		gLog.Println(LvERROR, "read log content error:", err)
+		return err
+	}
+	rsp := ReportLogRsp{}
+	rsp.Content = string(buff[:readLength])
+	rsp.FileName = req.FileName
+	rsp.Total = fi.Size()
+	rsp.Len = req.Len
+	return GNetwork.write(MsgReport, MsgPushReportLog, &rsp)
+}
+
+func handleReportGoroutine() (err error) {
+	gLog.Println(LvDEBUG, "handleReportGoroutine")
+	buf := make([]byte, 1024*128)
+	stackLen := runtime.Stack(buf, true)
+	return GNetwork.write(MsgReport, MsgPushReportLog, string(buf[:stackLen]))
+}
+
+func handleCheckRemoteService(msg []byte) (err error) {
+	gLog.Println(LvDEBUG, "handleCheckRemoteService")
+	req := CheckRemoteService{}
+	if err = json.Unmarshal(msg[openP2PHeaderSize:], &req); err != nil {
+		gLog.Printf(LvERROR, "wrong %v:%s  %s", reflect.TypeOf(req), err, string(msg[openP2PHeaderSize:]))
+		return err
+	}
+	rsp := PushRsp{Error: 0}
+	conn, err := net.DialTimeout("tcp", fmt.Sprintf("%s:%d", req.Host, req.Port), time.Second*3)
+	if err != nil {
+		rsp.Error = 1
+		rsp.Detail = ErrRemoteServiceUnable.Error()
+	} else {
+		conn.Close()
+	}
+	return GNetwork.write(MsgReport, MsgReportResponse, rsp)
+}

core/holepunch.go

@@ -3,15 +3,15 @@ package openp2p
 import (
 	"bytes"
 	"encoding/binary"
+	"encoding/json"
 	"fmt"
 	"math/rand"
 	"net"
-	"sync"
 	"time"
 )
 
 func handshakeC2C(t *P2PTunnel) (err error) {
-	gLog.Printf(LvDEBUG, "handshakeC2C %s:%d:%d to %s:%d", t.pn.config.Node, t.coneLocalPort, t.coneNatPort, t.config.peerIP, t.config.peerConeNatPort)
+	gLog.Printf(LvDEBUG, "handshakeC2C %s:%d:%d to %s:%d", gConf.Network.Node, t.coneLocalPort, t.coneNatPort, t.config.peerIP, t.config.peerConeNatPort)
 	defer gLog.Printf(LvDEBUG, "handshakeC2C end")
 	conn, err := net.ListenUDP("udp", t.la)
 	if err != nil {
@@ -23,34 +23,31 @@ func handshakeC2C(t *P2PTunnel) (err error) {
 		gLog.Println(LvDEBUG, "handshakeC2C write MsgPunchHandshake error:", err)
 		return err
 	}
-	ra, head, _, _, err := UDPRead(conn, 5000)
+	ra, head, buff, _, err := UDPRead(conn, HandshakeTimeout)
 	if err != nil {
-		time.Sleep(time.Millisecond * 200)
+		gLog.Println(LvDEBUG, "handshakeC2C read MsgPunchHandshake error:", err)
-		gLog.Println(LvDEBUG, err, ", return this error when ip was not reachable, retry read")
+		return err
-		ra, head, _, _, err = UDPRead(conn, 5000)
-		if err != nil {
-			gLog.Println(LvDEBUG, "handshakeC2C read MsgPunchHandshake error:", err)
-			return err
-		}
 	}
 	t.ra, _ = net.ResolveUDPAddr("udp", ra.String())
-	// cone server side
+	var tunnelID uint64
-	if head.MainType == MsgP2P && head.SubType == MsgPunchHandshake {
+	if len(buff) > openP2PHeaderSize {
+		req := P2PHandshakeReq{}
+		if err := json.Unmarshal(buff[openP2PHeaderSize:openP2PHeaderSize+int(head.DataLen)], &req); err == nil {
+			tunnelID = req.ID
+		}
+	} else { // compatible with old version
+		tunnelID = t.id
+	}
+	if head.MainType == MsgP2P && head.SubType == MsgPunchHandshake && tunnelID == t.id {
 		gLog.Printf(LvDEBUG, "read %d handshake ", t.id)
 		UDPWrite(conn, t.ra, MsgP2P, MsgPunchHandshakeAck, P2PHandshakeReq{ID: t.id})
-		_, head, _, _, err = UDPRead(conn, 5000)
+		_, head, _, _, err = UDPRead(conn, HandshakeTimeout)
 		if err != nil {
 			gLog.Println(LvDEBUG, "handshakeC2C write MsgPunchHandshakeAck error", err)
 			return err
 		}
-		if head.MainType == MsgP2P && head.SubType == MsgPunchHandshakeAck {
-			gLog.Printf(LvDEBUG, "read %d handshake ack ", t.id)
-			gLog.Printf(LvINFO, "handshakeC2C ok")
-			return nil
-		}
 	}
-	// cone client side will only read handshake ack
+	if head.MainType == MsgP2P && head.SubType == MsgPunchHandshakeAck && tunnelID == t.id {
-	if head.MainType == MsgP2P && head.SubType == MsgPunchHandshakeAck {
 		gLog.Printf(LvDEBUG, "read %d handshake ack ", t.id)
 		_, err = UDPWrite(conn, t.ra, MsgP2P, MsgPunchHandshakeAck, P2PHandshakeReq{ID: t.id})
 		if err != nil {
@@ -65,8 +62,12 @@ func handshakeC2C(t *P2PTunnel) (err error) {
 func handshakeC2S(t *P2PTunnel) error {
 	gLog.Printf(LvDEBUG, "handshakeC2S start")
 	defer gLog.Printf(LvDEBUG, "handshakeC2S end")
-	// even if read timeout, continue handshake
+	if !buildTunnelMtx.TryLock() {
-	t.pn.read(t.config.PeerNode, MsgPush, MsgPushHandshakeStart, SymmetricHandshakeAckTimeout)
+		// time.Sleep(time.Second * 3)
+		return ErrBuildTunnelBusy
+	}
+	defer buildTunnelMtx.Unlock()
+	startTime := time.Now()
 	r := rand.New(rand.NewSource(time.Now().UnixNano()))
 	randPorts := r.Perm(65532)
 	conn, err := net.ListenUDP("udp", t.la)
@@ -74,11 +75,11 @@ func handshakeC2S(t *P2PTunnel) error {
 		return err
 	}
 	defer conn.Close()
+
 	go func() error {
 		gLog.Printf(LvDEBUG, "send symmetric handshake to %s from %d:%d start", t.config.peerIP, t.coneLocalPort, t.coneNatPort)
 		for i := 0; i < SymmetricHandshakeNum; i++ {
-			// TODO: auto calc cost time
+			// time.Sleep(SymmetricHandshakeInterval)
-			time.Sleep(SymmetricHandshakeInterval)
 			dst, err := net.ResolveUDPAddr("udp", fmt.Sprintf("%s:%d", t.config.peerIP, randPorts[i]+2))
 			if err != nil {
 				return err
@@ -92,93 +93,161 @@ func handshakeC2S(t *P2PTunnel) error {
 		gLog.Println(LvDEBUG, "send symmetric handshake end")
 		return nil
 	}()
-	deadline := time.Now().Add(SymmetricHandshakeAckTimeout)
+	err = conn.SetReadDeadline(time.Now().Add(HandshakeTimeout))
-	err = conn.SetReadDeadline(deadline)
 	if err != nil {
 		gLog.Println(LvERROR, "SymmetricHandshakeAckTimeout SetReadDeadline error")
 		return err
 	}
 	// read response of the punching hole ok port
-	result := make([]byte, 1024)
+	buff := make([]byte, 1024)
-	_, dst, err := conn.ReadFrom(result)
+	_, dst, err := conn.ReadFrom(buff)
 	if err != nil {
 		gLog.Println(LvERROR, "handshakeC2S wait timeout")
 		return err
 	}
 	head := &openP2PHeader{}
-	err = binary.Read(bytes.NewReader(result[:openP2PHeaderSize]), binary.LittleEndian, head)
+	err = binary.Read(bytes.NewReader(buff[:openP2PHeaderSize]), binary.LittleEndian, head)
 	if err != nil {
 		gLog.Println(LvERROR, "parse p2pheader error:", err)
 		return err
 	}
 	t.ra, _ = net.ResolveUDPAddr("udp", dst.String())
-	if head.MainType == MsgP2P && head.SubType == MsgPunchHandshakeAck {
+	var tunnelID uint64
-		gLog.Printf(LvDEBUG, "handshakeC2S read %d handshake ack %s", t.id, dst.String())
+	if len(buff) > openP2PHeaderSize {
-		_, err = UDPWrite(conn, dst, MsgP2P, MsgPunchHandshakeAck, P2PHandshakeReq{ID: t.id})
+		req := P2PHandshakeReq{}
-		return err
+		if err := json.Unmarshal(buff[openP2PHeaderSize:openP2PHeaderSize+int(head.DataLen)], &req); err == nil {
+			tunnelID = req.ID
+		}
+	} else { // compatible with old version
+		tunnelID = t.id
 	}
-	gLog.Printf(LvINFO, "handshakeC2S ok")
+	if head.MainType == MsgP2P && head.SubType == MsgPunchHandshake && tunnelID == t.id {
+		gLog.Printf(LvDEBUG, "handshakeC2S read %d handshake ", t.id)
+		UDPWrite(conn, t.ra, MsgP2P, MsgPunchHandshakeAck, P2PHandshakeReq{ID: t.id})
+		for {
+			_, head, buff, _, err = UDPRead(conn, HandshakeTimeout)
+			if err != nil {
+				gLog.Println(LvDEBUG, "handshakeC2S handshake error")
+				return err
+			}
+			var tunnelID uint64
+			if len(buff) > openP2PHeaderSize {
+				req := P2PHandshakeReq{}
+				if err := json.Unmarshal(buff[openP2PHeaderSize:openP2PHeaderSize+int(head.DataLen)], &req); err == nil {
+					tunnelID = req.ID
+				}
+			} else { // compatible with old version
+				tunnelID = t.id
+			}
+			// waiting ack
+			if head.MainType == MsgP2P && head.SubType == MsgPunchHandshakeAck && tunnelID == t.id {
+				break
+			}
+		}
+	}
+	if head.MainType == MsgP2P && head.SubType == MsgPunchHandshakeAck {
+		gLog.Printf(LvDEBUG, "handshakeC2S read %d handshake ack %s", t.id, t.ra.String())
+		_, err = UDPWrite(conn, t.ra, MsgP2P, MsgPunchHandshakeAck, P2PHandshakeReq{ID: t.id})
+		return err
+	} else {
+		gLog.Println(LvDEBUG, "handshakeS2C read msg but not MsgPunchHandshakeAck")
+	}
+	gLog.Printf(LvINFO, "handshakeC2S ok. cost %d ms", time.Since(startTime)/time.Millisecond)
 	return nil
 }
 
 func handshakeS2C(t *P2PTunnel) error {
 	gLog.Printf(LvDEBUG, "handshakeS2C start")
 	defer gLog.Printf(LvDEBUG, "handshakeS2C end")
+	if !buildTunnelMtx.TryLock() {
+		// time.Sleep(time.Second * 3)
+		return ErrBuildTunnelBusy
+	}
+	defer buildTunnelMtx.Unlock()
+	startTime := time.Now()
 	gotCh := make(chan *net.UDPAddr, 5)
 	// sequencely udp send handshake, do not parallel send
 	gLog.Printf(LvDEBUG, "send symmetric handshake to %s:%d start", t.config.peerIP, t.config.peerConeNatPort)
 	gotIt := false
-	gotMtx := sync.Mutex{}
 	for i := 0; i < SymmetricHandshakeNum; i++ {
-		// TODO: auto calc cost time
+		// time.Sleep(SymmetricHandshakeInterval)
-		time.Sleep(SymmetricHandshakeInterval)
 		go func(t *P2PTunnel) error {
-			conn, err := net.ListenUDP("udp", nil)
+			conn, err := net.ListenUDP("udp", nil) // TODO: system allocated port really random?
 			if err != nil {
 				gLog.Printf(LvDEBUG, "listen error")
 				return err
 			}
 			defer conn.Close()
 			UDPWrite(conn, t.ra, MsgP2P, MsgPunchHandshake, P2PHandshakeReq{ID: t.id})
-			_, head, _, _, err := UDPRead(conn, 10000)
+			_, head, buff, _, err := UDPRead(conn, HandshakeTimeout)
 			if err != nil {
 				// gLog.Println(LevelDEBUG, "one of the handshake error:", err)
 				return err
 			}
-			gotMtx.Lock()
-			defer gotMtx.Unlock()
 			if gotIt {
 				return nil
 			}
-			gotIt = true
+			var tunnelID uint64
-			t.la, _ = net.ResolveUDPAddr("udp", conn.LocalAddr().String())
+			if len(buff) >= openP2PHeaderSize+8 {
-			if head.MainType == MsgP2P && head.SubType == MsgPunchHandshake {
+				req := P2PHandshakeReq{}
+				if err := json.Unmarshal(buff[openP2PHeaderSize:openP2PHeaderSize+int(head.DataLen)], &req); err == nil {
+					tunnelID = req.ID
+				}
+			} else { // compatible with old version
+				tunnelID = t.id
+			}
+
+			if head.MainType == MsgP2P && head.SubType == MsgPunchHandshake && tunnelID == t.id {
 				gLog.Printf(LvDEBUG, "handshakeS2C read %d handshake ", t.id)
 				UDPWrite(conn, t.ra, MsgP2P, MsgPunchHandshakeAck, P2PHandshakeReq{ID: t.id})
-				_, head, _, _, err = UDPRead(conn, 5000)
+				// may read several MsgPunchHandshake
-				if err != nil {
+				for {
-					gLog.Println(LvDEBUG, "handshakeS2C handshake error")
+					_, head, buff, _, err = UDPRead(conn, HandshakeTimeout)
-					return err
+					if err != nil {
-				}
+						gLog.Println(LvDEBUG, "handshakeS2C handshake error")
-				if head.MainType == MsgP2P && head.SubType == MsgPunchHandshakeAck {
+						return err
-					gLog.Printf(LvDEBUG, "handshakeS2C read %d handshake ack %s", t.id, conn.LocalAddr().String())
+					}
-					gotCh <- t.la
+					if len(buff) > openP2PHeaderSize {
-					return nil
+						req := P2PHandshakeReq{}
+						if err := json.Unmarshal(buff[openP2PHeaderSize:openP2PHeaderSize+int(head.DataLen)], &req); err == nil {
+							tunnelID = req.ID
+						}
+					} else { // compatible with old version
+						tunnelID = t.id
+					}
+					if head.MainType == MsgP2P && head.SubType == MsgPunchHandshakeAck && tunnelID == t.id {
+						break
+					} else {
+						gLog.Println(LvDEBUG, "handshakeS2C read msg but not MsgPunchHandshakeAck")
+					}
 				}
 			}
+			if head.MainType == MsgP2P && head.SubType == MsgPunchHandshakeAck {
+				gLog.Printf(LvDEBUG, "handshakeS2C read %d handshake ack %s", t.id, conn.LocalAddr().String())
+				UDPWrite(conn, t.ra, MsgP2P, MsgPunchHandshakeAck, P2PHandshakeReq{ID: t.id})
+				gotIt = true
+				la, _ := net.ResolveUDPAddr("udp", conn.LocalAddr().String())
+				gotCh <- la
+				return nil
+			} else {
+				gLog.Println(LvDEBUG, "handshakeS2C read msg but not MsgPunchHandshakeAck")
+			}
 			return nil
 		}(t)
 	}
 	gLog.Printf(LvDEBUG, "send symmetric handshake end")
-	gLog.Println(LvDEBUG, "handshakeS2C ready, notify peer connect")
+	if compareVersion(t.config.peerVersion, SymmetricSimultaneouslySendVersion) < 0 { // compatible with old client
-	t.pn.push(t.config.PeerNode, MsgPushHandshakeStart, TunnelMsg{ID: t.id})
+		gLog.Println(LvDEBUG, "handshakeS2C ready, notify peer connect")
+		t.pn.push(t.config.PeerNode, MsgPushHandshakeStart, TunnelMsg{ID: t.id})
+	}
 
 	select {
-	case <-time.After(SymmetricHandshakeAckTimeout):
+	case <-time.After(HandshakeTimeout):
-		return fmt.Errorf("wait handshake failed")
+		return fmt.Errorf("wait handshake timeout")
 	case la := <-gotCh:
+		t.la = la
 		gLog.Println(LvDEBUG, "symmetric handshake ok", la)
-		gLog.Printf(LvINFO, "handshakeS2C ok")
+		gLog.Printf(LvINFO, "handshakeS2C ok. cost %dms", time.Since(startTime)/time.Millisecond)
 	}
 	return nil
 }

core/install.go

@@ -1,129 +1,124 @@
-package openp2p
+package openp2p
-
+
-import (
+import (
-	"fmt"
+	"fmt"
-	"io"
+	"io"
-	"os"
+	"os"
-	"os/exec"
+	"os/exec"
-	"path/filepath"
+	"path/filepath"
-	"strings"
+	"strings"
-	"time"
+	"time"
-)
+)
-
+
-// examples:
+func install() {
-// listen:
+	gLog.Println(LvINFO, "openp2p start. version: ", OpenP2PVersion)
-// ./openp2p install -node hhd1207-222 -token YOUR-TOKEN -sharebandwidth 0
+	gLog.Println(LvINFO, "Contact: QQ group 16947733, Email [email protected]")
-// listen and build p2papp:
+	gLog.Println(LvINFO, "install start")
-// ./openp2p install -node hhd1207-222 -token YOUR-TOKEN -sharebandwidth 0 -peernode hhdhome-n1 -dstip 127.0.0.1 -dstport 50022 -protocol tcp -srcport 22
+	defer gLog.Println(LvINFO, "install end")
-func install() {
+	// auto uninstall
-	gLog.Println(LvINFO, "openp2p start. version: ", OpenP2PVersion)
+	err := os.MkdirAll(defaultInstallPath, 0775)
-	gLog.Println(LvINFO, "Contact: QQ group 16947733, Email [email protected]")
+
-	gLog.Println(LvINFO, "install start")
+	if err != nil {
-	defer gLog.Println(LvINFO, "install end")
+		gLog.Printf(LvERROR, "MkdirAll %s error:%s", defaultInstallPath, err)
-	// auto uninstall
+		return
-	err := os.MkdirAll(defaultInstallPath, 0775)
+	}
-
+	err = os.Chdir(defaultInstallPath)
-	if err != nil {
+	if err != nil {
-		gLog.Printf(LvERROR, "MkdirAll %s error:%s", defaultInstallPath, err)
+		gLog.Println(LvERROR, "cd error:", err)
-		return
+		return
-	}
+	}
-	err = os.Chdir(defaultInstallPath)
+
-	if err != nil {
+	uninstall()
-		gLog.Println(LvERROR, "cd error:", err)
+	// save config file
-		return
+	parseParams("install", "")
-	}
+	targetPath := filepath.Join(defaultInstallPath, defaultBinName)
-
+	d := daemon{}
-	uninstall()
+	// copy files
-	// save config file
+
-	parseParams("install")
+	binPath, _ := os.Executable()
-	targetPath := filepath.Join(defaultInstallPath, defaultBinName)
+	src, errFiles := os.Open(binPath) // can not use args[0], on Windows call openp2p is ok(=openp2p.exe)
-	d := daemon{}
+	if errFiles != nil {
-	// copy files
+		gLog.Printf(LvERROR, "os.OpenFile %s error:%s", os.Args[0], errFiles)
-
+		return
-	binPath, _ := os.Executable()
+	}
-	src, errFiles := os.Open(binPath) // can not use args[0], on Windows call openp2p is ok(=openp2p.exe)
+
-	if errFiles != nil {
+	dst, errFiles := os.OpenFile(targetPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0775)
-		gLog.Printf(LvERROR, "os.OpenFile %s error:%s", os.Args[0], errFiles)
+	if errFiles != nil {
-		return
+		gLog.Printf(LvERROR, "os.OpenFile %s error:%s", targetPath, errFiles)
-	}
+		return
-
+	}
-	dst, errFiles := os.OpenFile(targetPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0775)
+
-	if errFiles != nil {
+	_, errFiles = io.Copy(dst, src)
-		gLog.Printf(LvERROR, "os.OpenFile %s error:%s", targetPath, errFiles)
+	if errFiles != nil {
-		return
+		gLog.Printf(LvERROR, "io.Copy error:%s", errFiles)
-	}
+		return
-
+	}
-	_, errFiles = io.Copy(dst, src)
+	src.Close()
-	if errFiles != nil {
+	dst.Close()
-		gLog.Printf(LvERROR, "io.Copy error:%s", errFiles)
+
-		return
+	// install system service
-	}
+	gLog.Println(LvINFO, "targetPath:", targetPath)
-	src.Close()
+	err = d.Control("install", targetPath, []string{"-d"})
-	dst.Close()
+	if err == nil {
-
+		gLog.Println(LvINFO, "install system service ok.")
-	// install system service
+	}
-	gLog.Println(LvINFO, "targetPath:", targetPath)
+	time.Sleep(time.Second * 2)
-	err = d.Control("install", targetPath, []string{"-d"})
+	err = d.Control("start", targetPath, []string{"-d"})
-	if err == nil {
+	if err != nil {
-		gLog.Println(LvINFO, "install system service ok.")
+		gLog.Println(LvERROR, "start openp2p service error:", err)
-	}
+	} else {
-	time.Sleep(time.Second * 2)
+		gLog.Println(LvINFO, "start openp2p service ok.")
-	err = d.Control("start", targetPath, []string{"-d"})
+	}
-	if err != nil {
+	gLog.Println(LvINFO, "Visit WebUI on https://console.openp2p.cn")
-		gLog.Println(LvERROR, "start openp2p service error:", err)
+}
-	} else {
+
-		gLog.Println(LvINFO, "start openp2p service ok.")
+func installByFilename() {
-	}
+	params := strings.Split(filepath.Base(os.Args[0]), "-")
-	gLog.Println(LvINFO, "Visit WebUI on https://console.openp2p.cn")
+	if len(params) < 4 {
-}
+		return
-
+	}
-func installByFilename() {
+	serverHost := params[1]
-	params := strings.Split(filepath.Base(os.Args[0]), "-")
+	token := params[2]
-	if len(params) < 4 {
+	gLog.Println(LvINFO, "install start")
-		return
+	targetPath := os.Args[0]
-	}
+	args := []string{"install"}
-	serverHost := params[1]
+	args = append(args, "-serverhost")
-	token := params[2]
+	args = append(args, serverHost)
-	gLog.Println(LvINFO, "install start")
+	args = append(args, "-token")
-	targetPath := os.Args[0]
+	args = append(args, token)
-	args := []string{"install"}
+	env := os.Environ()
-	args = append(args, "-serverhost")
+	cmd := exec.Command(targetPath, args...)
-	args = append(args, serverHost)
+	cmd.Stdout = os.Stdout
-	args = append(args, "-token")
+	cmd.Stderr = os.Stderr
-	args = append(args, token)
+	cmd.Stdin = os.Stdin
-	env := os.Environ()
+	cmd.Env = env
-	cmd := exec.Command(targetPath, args...)
+	err := cmd.Run()
-	cmd.Stdout = os.Stdout
+	if err != nil {
-	cmd.Stderr = os.Stderr
+		gLog.Println(LvERROR, "install by filename, start process error:", err)
-	cmd.Stdin = os.Stdin
+		return
-	cmd.Env = env
+	}
-	err := cmd.Run()
+	gLog.Println(LvINFO, "install end")
-	if err != nil {
+	gLog.Println(LvINFO, "Visit WebUI on https://console.openp2p.cn")
-		gLog.Println(LvERROR, "install by filename, start process error:", err)
+	fmt.Println("Press the Any Key to exit")
-		return
+	fmt.Scanln()
-	}
+	os.Exit(0)
-	gLog.Println(LvINFO, "install end")
+}
-	gLog.Println(LvINFO, "Visit WebUI on https://console.openp2p.cn")
+func uninstall() {
-	fmt.Println("Press the Any Key to exit")
+	gLog.Println(LvINFO, "uninstall start")
-	fmt.Scanln()
+	defer gLog.Println(LvINFO, "uninstall end")
-	os.Exit(0)
+	d := daemon{}
-}
+	err := d.Control("stop", "", nil)
-func uninstall() {
+	if err != nil { // service maybe not install
-	gLog.Println(LvINFO, "uninstall start")
+		return
-	defer gLog.Println(LvINFO, "uninstall end")
+	}
-	d := daemon{}
+	err = d.Control("uninstall", "", nil)
-	err := d.Control("stop", "", nil)
+	if err != nil {
-	if err != nil { // service maybe not install
+		gLog.Println(LvERROR, "uninstall system service error:", err)
-		return
+	} else {
-	}
+		gLog.Println(LvINFO, "uninstall system service ok.")
-	err = d.Control("uninstall", "", nil)
+	}
-	if err != nil {
+	binPath := filepath.Join(defaultInstallPath, defaultBinName)
-		gLog.Println(LvERROR, "uninstall system service error:", err)
+	os.Remove(binPath + "0")
-	} else {
+	os.Remove(binPath)
-		gLog.Println(LvINFO, "uninstall system service ok.")
+	// os.RemoveAll(defaultInstallPath)  // reserve config.json
-	}
+}
-	binPath := filepath.Join(defaultInstallPath, defaultBinName)
-	os.Remove(binPath + "0")
-	os.Remove(binPath)
-	// os.RemoveAll(defaultInstallPath)  // reserve config.json
-}

core/iptables.go

@@ -0,0 +1,74 @@
+package openp2p
+
+import (
+	"log"
+	"os/exec"
+	"runtime"
+)
+
+func allowTunForward() {
+	if runtime.GOOS != "linux" { // only support Linux
+		return
+	}
+	exec.Command("sh", "-c", `iptables -t filter -D FORWARD -i optun -j ACCEPT`).Run()
+	exec.Command("sh", "-c", `iptables -t filter -D FORWARD -o optun -j ACCEPT`).Run()
+	err := exec.Command("sh", "-c", `iptables -t filter -I FORWARD -i optun -j ACCEPT`).Run()
+	if err != nil {
+		log.Println("allow foward in error:", err)
+	}
+	err = exec.Command("sh", "-c", `iptables -t filter -I FORWARD -o optun -j ACCEPT`).Run()
+	if err != nil {
+		log.Println("allow foward out error:", err)
+	}
+}
+
+func clearSNATRule() {
+	if runtime.GOOS != "linux" {
+		return
+	}
+	execCommand("iptables", true, "-t", "nat", "-D", "POSTROUTING", "-j", "OPSDWAN")
+	execCommand("iptables", true, "-t", "nat", "-F", "OPSDWAN")
+	execCommand("iptables", true, "-t", "nat", "-X", "OPSDWAN")
+}
+
+func initSNATRule(localNet string) {
+	if runtime.GOOS != "linux" {
+		return
+	}
+	clearSNATRule()
+
+	err := execCommand("iptables", true, "-t", "nat", "-N", "OPSDWAN")
+	if err != nil {
+		log.Println("iptables new sdwan chain error:", err)
+		return
+	}
+	err = execCommand("iptables", true, "-t", "nat", "-A", "POSTROUTING", "-j", "OPSDWAN")
+	if err != nil {
+		log.Println("iptables append postrouting error:", err)
+		return
+	}
+	err = execCommand("iptables", true, "-t", "nat", "-A", "OPSDWAN",
+		"-o", "optun", "!", "-s", localNet, "-j", "MASQUERADE")
+	if err != nil {
+		log.Println("add optun snat error:", err)
+		return
+	}
+	err = execCommand("iptables", true, "-t", "nat", "-A", "OPSDWAN", "!", "-o", "optun",
+		"-s", localNet, "-j", "MASQUERADE")
+	if err != nil {
+		log.Println("add optun snat error:", err)
+		return
+	}
+}
+
+func addSNATRule(target string) {
+	if runtime.GOOS != "linux" {
+		return
+	}
+	err := execCommand("iptables", true, "-t", "nat", "-A", "OPSDWAN", "!", "-o", "optun",
+		"-s", target, "-j", "MASQUERADE")
+	if err != nil {
+		log.Println("iptables add optun snat error:", err)
+		return
+	}
+}

core/iptree.go

@@ -0,0 +1,179 @@
+package openp2p
+
+import (
+	"bytes"
+	"encoding/binary"
+	"fmt"
+	"log"
+	"net"
+	"strings"
+	"sync"
+
+	"github.com/emirpasic/gods/trees/avltree"
+	"github.com/emirpasic/gods/utils"
+)
+
+type IPTree struct {
+	tree    *avltree.Tree
+	treeMtx sync.RWMutex
+}
+type IPTreeValue struct {
+	maxIP uint32
+	v     interface{}
+}
+
+// TODO: deal interset
+func (iptree *IPTree) DelIntIP(minIP uint32, maxIP uint32) {
+	iptree.tree.Remove(minIP)
+}
+
+// add 120k cost 0.5s
+func (iptree *IPTree) AddIntIP(minIP uint32, maxIP uint32, v interface{}) bool {
+	if minIP > maxIP {
+		return false
+	}
+	iptree.treeMtx.Lock()
+	defer iptree.treeMtx.Unlock()
+	newMinIP := minIP
+	newMaxIP := maxIP
+	cur := iptree.tree.Root
+	for {
+		if cur == nil {
+			break
+		}
+		tv := cur.Value.(*IPTreeValue)
+		curMinIP := cur.Key.(uint32)
+
+		// newNode all in existNode, treat as inserted.
+		if newMinIP >= curMinIP && newMaxIP <= tv.maxIP {
+			return true
+		}
+		// has no interset
+		if newMinIP > tv.maxIP {
+			cur = cur.Children[1]
+			continue
+		}
+		if newMaxIP < curMinIP {
+			cur = cur.Children[0]
+			continue
+		}
+		// has interset, rm it and Add the new merged ip segment
+		iptree.tree.Remove(curMinIP)
+		if curMinIP < newMinIP {
+			newMinIP = curMinIP
+		}
+		if tv.maxIP > newMaxIP {
+			newMaxIP = tv.maxIP
+		}
+		cur = iptree.tree.Root
+	}
+	//  put in the tree
+	iptree.tree.Put(newMinIP, &IPTreeValue{newMaxIP, v})
+	return true
+}
+
+func (iptree *IPTree) Add(minIPStr string, maxIPStr string, v interface{}) bool {
+	var minIP, maxIP uint32
+	binary.Read(bytes.NewBuffer(net.ParseIP(minIPStr).To4()), binary.BigEndian, &minIP)
+	binary.Read(bytes.NewBuffer(net.ParseIP(maxIPStr).To4()), binary.BigEndian, &maxIP)
+	return iptree.AddIntIP(minIP, maxIP, v)
+}
+
+func (iptree *IPTree) Del(minIPStr string, maxIPStr string) {
+	var minIP, maxIP uint32
+	binary.Read(bytes.NewBuffer(net.ParseIP(minIPStr).To4()), binary.BigEndian, &minIP)
+	binary.Read(bytes.NewBuffer(net.ParseIP(maxIPStr).To4()), binary.BigEndian, &maxIP)
+	iptree.DelIntIP(minIP, maxIP)
+}
+
+func (iptree *IPTree) Contains(ipStr string) bool {
+	var ip uint32
+	binary.Read(bytes.NewBuffer(net.ParseIP(ipStr).To4()), binary.BigEndian, &ip)
+	_, ok := iptree.Load(ip)
+	return ok
+}
+
+func IsLocalhost(ipStr string) bool {
+	if ipStr == "localhost" || ipStr == "127.0.0.1" || ipStr == "::1" {
+		return true
+	}
+	return false
+}
+
+func (iptree *IPTree) Load(ip uint32) (interface{}, bool) {
+	iptree.treeMtx.RLock()
+	defer iptree.treeMtx.RUnlock()
+	if iptree.tree == nil {
+		return nil, false
+	}
+	n := iptree.tree.Root
+	for n != nil {
+		tv := n.Value.(*IPTreeValue)
+		curMinIP := n.Key.(uint32)
+		switch {
+		case ip >= curMinIP && ip <= tv.maxIP: // hit
+			return tv.v, true
+		case ip < curMinIP:
+			n = n.Children[0]
+		default:
+			n = n.Children[1]
+		}
+	}
+	return nil, false
+}
+
+func (iptree *IPTree) Size() int {
+	iptree.treeMtx.RLock()
+	defer iptree.treeMtx.RUnlock()
+	return iptree.tree.Size()
+}
+
+func (iptree *IPTree) Print() {
+	iptree.treeMtx.RLock()
+	defer iptree.treeMtx.RUnlock()
+	log.Println("size:", iptree.Size())
+	log.Println(iptree.tree.String())
+}
+
+func (iptree *IPTree) Clear() {
+	iptree.treeMtx.Lock()
+	defer iptree.treeMtx.Unlock()
+	iptree.tree.Clear()
+}
+
+// input format 127.0.0.1,192.168.1.0/24,10.1.1.30-10.1.1.50
+// 127.0.0.1
+// 192.168.1.0/24
+// 192.168.1.1-192.168.1.10
+func NewIPTree(ips string) *IPTree {
+	iptree := &IPTree{
+		tree: avltree.NewWith(utils.UInt32Comparator),
+	}
+	ipArr := strings.Split(ips, ",")
+	for _, ip := range ipArr {
+		if strings.Contains(ip, "/") { // x.x.x.x/24
+			_, ipNet, err := net.ParseCIDR(ip)
+			if err != nil {
+				fmt.Println("Error parsing CIDR:", err)
+				continue
+			}
+			minIP := ipNet.IP.Mask(ipNet.Mask).String()
+			maxIP := calculateMaxIP(ipNet).String()
+			iptree.Add(minIP, maxIP, nil)
+		} else if strings.Contains(ip, "-") { // x.x.x.x-y.y.y.y
+			minAndMax := strings.Split(ip, "-")
+			iptree.Add(minAndMax[0], minAndMax[1], nil)
+		} else { // single ip
+			iptree.Add(ip, ip, nil)
+		}
+	}
+	return iptree
+}
+func calculateMaxIP(ipNet *net.IPNet) net.IP {
+	maxIP := make(net.IP, len(ipNet.IP))
+	copy(maxIP, ipNet.IP)
+	for i := range maxIP {
+		maxIP[i] |= ^ipNet.Mask[i]
+	}
+	return maxIP
+}

core/iptree_test.go

@@ -0,0 +1,174 @@
+package openp2p
+
+import (
+	"bytes"
+	"encoding/binary"
+	"net"
+	"testing"
+	"time"
+)
+
+func wrapTestContains(t *testing.T, iptree *IPTree, ip string, result bool) {
+	if iptree.Contains(ip) == result {
+		// t.Logf("compare version %s %s ok\n", v1, v2)
+	} else {
+		t.Errorf("test %s fail\n", ip)
+	}
+}
+func wrapBenchmarkContains(t *testing.B, iptree *IPTree, ip string, result bool) {
+	if iptree.Contains(ip) == result {
+		// t.Logf("compare version %s %s ok\n", v1, v2)
+	} else {
+		t.Errorf("test %s fail\n", ip)
+	}
+}
+
+func TestAllInputFormat(t *testing.T) {
+	iptree := NewIPTree("219.137.185.70,127.0.0.1,127.0.0.0/8,192.168.1.0/24,192.168.3.100-192.168.3.255,192.168.100.0-192.168.200.255")
+	wrapTestContains(t, iptree, "127.0.0.1", true)
+	wrapTestContains(t, iptree, "127.0.0.2", true)
+	wrapTestContains(t, iptree, "127.1.1.1", true)
+	wrapTestContains(t, iptree, "219.137.185.70", true)
+	wrapTestContains(t, iptree, "219.137.185.71", false)
+	wrapTestContains(t, iptree, "192.168.1.2", true)
+	wrapTestContains(t, iptree, "192.168.2.2", false)
+	wrapTestContains(t, iptree, "192.168.3.1", false)
+	wrapTestContains(t, iptree, "192.168.3.100", true)
+	wrapTestContains(t, iptree, "192.168.3.255", true)
+	wrapTestContains(t, iptree, "192.168.150.1", true)
+	wrapTestContains(t, iptree, "192.168.250.1", false)
+}
+
+func TestSingleIP(t *testing.T) {
+	iptree := NewIPTree("")
+	iptree.Add("219.137.185.70", "219.137.185.70", nil)
+	wrapTestContains(t, iptree, "219.137.185.70", true)
+	wrapTestContains(t, iptree, "219.137.185.71", false)
+}
+
+func TestWrongSegment(t *testing.T) {
+	iptree := NewIPTree("")
+	inserted := iptree.Add("87.251.75.0", "82.251.75.255", nil)
+	if inserted {
+		t.Errorf("TestWrongSegment failed\n")
+	}
+}
+
+func TestSegment2(t *testing.T) {
+	iptree := NewIPTree("")
+	iptree.Clear()
+	iptree.Add("10.1.5.50", "10.1.5.100", nil)
+	iptree.Add("10.1.1.50", "10.1.1.100", nil)
+	iptree.Add("10.1.2.50", "10.1.2.100", nil)
+	iptree.Add("10.1.6.50", "10.1.6.100", nil)
+	iptree.Add("10.1.7.50", "10.1.7.100", nil)
+	iptree.Add("10.1.3.50", "10.1.3.100", nil)
+	iptree.Add("10.1.1.1", "10.1.1.10", nil)    // no interset
+	iptree.Add("10.1.1.200", "10.1.1.250", nil) // no interset
+	iptree.Print()
+
+	iptree.Add("10.1.1.80", "10.1.1.90", nil) // all in
+	iptree.Add("10.1.1.40", "10.1.1.60", nil) // interset
+	iptree.Print()
+	iptree.Add("10.1.1.90", "10.1.1.110", nil) // interset
+	iptree.Print()
+	t.Logf("ipTree size:%d\n", iptree.Size())
+	wrapTestContains(t, iptree, "10.1.1.40", true)
+	wrapTestContains(t, iptree, "10.1.5.50", true)
+	wrapTestContains(t, iptree, "10.1.6.50", true)
+	wrapTestContains(t, iptree, "10.1.7.50", true)
+	wrapTestContains(t, iptree, "10.1.2.50", true)
+	wrapTestContains(t, iptree, "10.1.3.50", true)
+	wrapTestContains(t, iptree, "10.1.1.60", true)
+	wrapTestContains(t, iptree, "10.1.1.90", true)
+	wrapTestContains(t, iptree, "10.1.1.110", true)
+	wrapTestContains(t, iptree, "10.1.1.250", true)
+	wrapTestContains(t, iptree, "10.1.2.60", true)
+	wrapTestContains(t, iptree, "10.1.100.30", false)
+	wrapTestContains(t, iptree, "10.1.200.30", false)
+
+	iptree.Add("10.0.0.0", "10.255.255.255", nil) // will merge all segment
+	iptree.Print()
+	if iptree.Size() != 1 {
+		t.Errorf("merge ip segment error\n")
+	}
+
+}
+
+func BenchmarkBuildipTree20k(t *testing.B) {
+	iptree := NewIPTree("")
+	iptree.Clear()
+	iptree.Add("10.1.5.50", "10.1.5.100", nil)
+	iptree.Add("10.1.1.50", "10.1.1.100", nil)
+	iptree.Add("10.1.2.50", "10.1.2.100", nil)
+	iptree.Add("10.1.6.50", "10.1.6.100", nil)
+	iptree.Add("10.1.7.50", "10.1.7.100", nil)
+	iptree.Add("10.1.3.50", "10.1.3.100", nil)
+	iptree.Add("10.1.1.1", "10.1.1.10", nil)    // no interset
+	iptree.Add("10.1.1.200", "10.1.1.250", nil) // no interset
+	iptree.Add("10.1.1.80", "10.1.1.90", nil)   // all in
+	iptree.Add("10.1.1.40", "10.1.1.60", nil)   // interset
+	iptree.Add("10.1.1.90", "10.1.1.110", nil)  // interset
+	var minIP uint32
+	binary.Read(bytes.NewBuffer(net.ParseIP("10.1.1.1").To4()), binary.BigEndian, &minIP)
+
+	// insert 10k block ip single
+	nodeNum := uint32(10000 * 1)
+	gap := uint32(10)
+	for i := minIP; i < minIP+nodeNum*gap; i += gap {
+		iptree.AddIntIP(i, i, nil)
+		// t.Logf("ipTree size:%d\n", iptree.Size())
+	}
+	binary.Read(bytes.NewBuffer(net.ParseIP("100.1.1.1").To4()), binary.BigEndian, &minIP)
+	// insert 100k block ip segment
+	for i := minIP; i < minIP+nodeNum*gap; i += gap {
+		iptree.AddIntIP(i, i+5, nil)
+	}
+	t.Logf("ipTree size:%d\n", iptree.Size())
+	iptree.Clear()
+	t.Logf("clear. ipTree size:%d\n", iptree.Size())
+}
+func BenchmarkQuery(t *testing.B) {
+	ts := time.Now()
+	iptree := NewIPTree("")
+	iptree.Clear()
+	iptree.Add("10.1.5.50", "10.1.5.100", nil)
+	iptree.Add("10.1.1.50", "10.1.1.100", nil)
+	iptree.Add("10.1.2.50", "10.1.2.100", nil)
+	iptree.Add("10.1.6.50", "10.1.6.100", nil)
+	iptree.Add("10.1.7.50", "10.1.7.100", nil)
+	iptree.Add("10.1.3.50", "10.1.3.100", nil)
+	iptree.Add("10.1.1.1", "10.1.1.10", nil)    // no interset
+	iptree.Add("10.1.1.200", "10.1.1.250", nil) // no interset
+	iptree.Add("10.1.1.80", "10.1.1.90", nil)   // all in
+	iptree.Add("10.1.1.40", "10.1.1.60", nil)   // interset
+	iptree.Add("10.1.1.90", "10.1.1.110", nil)  // interset
+	var minIP uint32
+	binary.Read(bytes.NewBuffer(net.ParseIP("10.1.1.1").To4()), binary.BigEndian, &minIP)
+
+	// insert 10k block ip single
+	nodeNum := uint32(10000 * 1000)
+	gap := uint32(10)
+	for i := minIP; i < minIP+nodeNum*gap; i += gap {
+		iptree.AddIntIP(i, i, nil)
+		// t.Logf("ipTree size:%d\n", iptree.Size())
+	}
+	binary.Read(bytes.NewBuffer(net.ParseIP("100.1.1.1").To4()), binary.BigEndian, &minIP)
+	// insert 100k block ip segment
+	for i := minIP; i < minIP+nodeNum*gap; i += gap {
+		iptree.AddIntIP(i, i+5, nil)
+	}
+	t.Logf("ipTree size:%d cost:%dms\n", iptree.Size(), time.Since(ts)/time.Millisecond)
+	ts = time.Now()
+	// t.ResetTimer()
+	queryNum := 100 * 10000
+	for i := 0; i < queryNum; i++ {
+		iptree.Load(minIP + uint32(i))
+		wrapBenchmarkContains(t, iptree, "10.1.5.55", true)
+		wrapBenchmarkContains(t, iptree, "10.1.1.1", true)
+		wrapBenchmarkContains(t, iptree, "10.1.5.200", false)
+		wrapBenchmarkContains(t, iptree, "200.1.1.1", false)
+	}
+	t.Logf("query num:%d cost:%dms\n", queryNum*4, time.Since(ts)/time.Millisecond)
+
+}

core/log.go

@@ -13,6 +13,7 @@ type LogLevel int
 var gLog *logger
 
 const (
+	LvDev   LogLevel = -1
 	LvDEBUG LogLevel = iota
 	LvINFO
 	LvWARN
@@ -32,13 +33,13 @@ func init() {
 	loglevel[LvINFO] = "INFO"
 	loglevel[LvWARN] = "WARN"
 	loglevel[LvERROR] = "ERROR"
+	loglevel[LvDev] = "Dev"
 
 }
 
 const (
-	LogFile = iota
+	LogFile    = 1
-	LogConsole
+	LogConsole = 1 << 1
-	LogFileAndConsole
 )
 
 type logger struct {
@@ -51,6 +52,7 @@ type logger struct {
 	pid        int
 	maxLogSize int64
 	mode       int
+	stdLogger  *log.Logger
 }
 
 func NewLogger(path string, filePrefix string, level LogLevel, maxLogSize int64, mode int) *logger {
@@ -67,13 +69,13 @@ func NewLogger(path string, filePrefix string, level LogLevel, maxLogSize int64,
 	os.MkdirAll(logdir, 0777)
 	for lv := range logFileNames {
 		logFilePath := logdir + filePrefix + logFileNames[lv]
-		f, err := os.OpenFile(logFilePath, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
+		f, err := os.OpenFile(logFilePath, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644)
 		if err != nil {
 			log.Fatal(err)
 		}
-		os.Chmod(logFilePath, 0666)
+		os.Chmod(logFilePath, 0644)
 		logfiles[lv] = f
-		loggers[lv] = log.New(f, "", log.LstdFlags)
+		loggers[lv] = log.New(f, "", log.LstdFlags|log.Lmicroseconds)
 	}
 	var le string
 	if runtime.GOOS == "windows" {
@@ -81,7 +83,8 @@ func NewLogger(path string, filePrefix string, level LogLevel, maxLogSize int64,
 	} else {
 		le = "\n"
 	}
-	pLog := &logger{loggers, logfiles, level, logdir, &sync.Mutex{}, le, os.Getpid(), maxLogSize, mode}
+	pLog := &logger{loggers, logfiles, level, logdir, &sync.Mutex{}, le, os.Getpid(), maxLogSize, mode, log.New(os.Stdout, "", 0)}
+	pLog.stdLogger.SetFlags(log.LstdFlags | log.Lmicroseconds)
 	go pLog.checkFile()
 	return pLog
 }
@@ -91,6 +94,13 @@ func (l *logger) setLevel(level LogLevel) {
 	defer l.mtx.Unlock()
 	l.level = level
 }
+
+func (l *logger) setMaxSize(size int64) {
+	l.mtx.Lock()
+	defer l.mtx.Unlock()
+	l.maxLogSize = size
+}
+
 func (l *logger) setMode(mode int) {
 	l.mtx.Lock()
 	defer l.mtx.Unlock()
@@ -119,7 +129,7 @@ func (l *logger) checkFile() {
 				backupPath := l.logDir + fname + ".0"
 				os.Remove(backupPath)
 				os.Rename(l.logDir+fname, backupPath)
-				newFile, e := os.OpenFile(l.logDir+fname, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
+				newFile, e := os.OpenFile(l.logDir+fname, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644)
 				if e == nil {
 					l.loggers[lv].SetOutput(newFile)
 					l.files[lv] = newFile
@@ -138,11 +148,11 @@ func (l *logger) Printf(level LogLevel, format string, params ...interface{}) {
 	}
 	pidAndLevel := []interface{}{l.pid, loglevel[level]}
 	params = append(pidAndLevel, params...)
-	if l.mode == LogFile || l.mode == LogFileAndConsole {
+	if l.mode&LogFile != 0 {
 		l.loggers[0].Printf("%d %s "+format+l.lineEnding, params...)
 	}
-	if l.mode == LogConsole || l.mode == LogFileAndConsole {
+	if l.mode&LogConsole != 0 {
-		log.Printf("%d %s "+format+l.lineEnding, params...)
+		l.stdLogger.Printf("%d %s "+format+l.lineEnding, params...)
 	}
 }
 
@@ -155,10 +165,10 @@ func (l *logger) Println(level LogLevel, params ...interface{}) {
 	pidAndLevel := []interface{}{l.pid, " ", loglevel[level], " "}
 	params = append(pidAndLevel, params...)
 	params = append(params, l.lineEnding)
-	if l.mode == LogFile || l.mode == LogFileAndConsole {
+	if l.mode&LogFile != 0 {
 		l.loggers[0].Print(params...)
 	}
-	if l.mode == LogConsole || l.mode == LogFileAndConsole {
+	if l.mode&LogConsole != 0 {
-		log.Print(params...)
+		l.stdLogger.Print(params...)
 	}
 }

core/nat.go

@@ -1,192 +1,189 @@
-package openp2p
+package openp2p
-
+
-import (
+import (
-	"encoding/json"
+	"encoding/json"
-	"fmt"
+	"fmt"
-	"log"
+	"math/rand"
-	"math/rand"
+	"net"
-	"net"
+	"strconv"
-	"strconv"
+	"strings"
-	"strings"
+	"time"
-	"sync"
+
-	"time"
+	reuse "github.com/openp2p-cn/go-reuseport"
-
+)
-	reuse "github.com/openp2p-cn/go-reuseport"
+
-)
+func natTCP(serverHost string, serverPort int) (publicIP string, publicPort int, localPort int) {
-
+	// dialer := &net.Dialer{
-func natTCP(serverHost string, serverPort int, localPort int) (publicIP string, publicPort int) {
+	// 	LocalAddr: &net.TCPAddr{
-	// dialer := &net.Dialer{
+	// 		IP:   net.ParseIP("0.0.0.0"),
-	// 	LocalAddr: &net.TCPAddr{
+	// 		Port: localPort,
-	// 		IP:   net.ParseIP("0.0.0.0"),
+	// 	},
-	// 		Port: localPort,
+	// }
-	// 	},
+	conn, err := reuse.DialTimeout("tcp4", fmt.Sprintf("%s:%d", "0.0.0.0", 0), fmt.Sprintf("%s:%d", serverHost, serverPort), NatTestTimeout)
-	// }
+	// conn, err := net.Dial("tcp4", fmt.Sprintf("%s:%d", serverHost, serverPort))
-	conn, err := reuse.DialTimeout("tcp4", fmt.Sprintf("%s:%d", "0.0.0.0", localPort), fmt.Sprintf("%s:%d", serverHost, serverPort), time.Second*5)
+	// log.Println(LvINFO, conn.LocalAddr())
-	// conn, err := net.Dial("tcp4", fmt.Sprintf("%s:%d", serverHost, serverPort))
+	if err != nil {
-	if err != nil {
+		fmt.Printf("Dial tcp4 %s:%d error:%s", serverHost, serverPort, err)
-		fmt.Printf("Dial tcp4 %s:%d error:%s", serverHost, serverPort, err)
+		return
-		return
+	}
-	}
+	defer conn.Close()
-	defer conn.Close()
+	localPort, _ = strconv.Atoi(strings.Split(conn.LocalAddr().String(), ":")[1])
-	_, wrerr := conn.Write([]byte("1"))
+	_, wrerr := conn.Write([]byte("1"))
-	if wrerr != nil {
+	if wrerr != nil {
-		fmt.Printf("Write error: %s\n", wrerr)
+		fmt.Printf("Write error: %s\n", wrerr)
-		return
+		return
-	}
+	}
-	b := make([]byte, 1000)
+	b := make([]byte, 1000)
-	conn.SetReadDeadline(time.Now().Add(time.Second * 5))
+	conn.SetReadDeadline(time.Now().Add(NatTestTimeout))
-	n, rderr := conn.Read(b)
+	n, rderr := conn.Read(b)
-	if rderr != nil {
+	if rderr != nil {
-		fmt.Printf("Read error: %s\n", rderr)
+		fmt.Printf("Read error: %s\n", rderr)
-		return
+		return
-	}
+	}
-	arr := strings.Split(string(b[:n]), ":")
+	arr := strings.Split(string(b[:n]), ":")
-	if len(arr) < 2 {
+	if len(arr) < 2 {
-		return
+		return
-	}
+	}
-	publicIP = arr[0]
+	publicIP = arr[0]
-	port, _ := strconv.ParseInt(arr[1], 10, 32)
+	port, _ := strconv.ParseInt(arr[1], 10, 32)
-	publicPort = int(port)
+	publicPort = int(port)
-	return
+	return
-
+
-}
+}
-func natTest(serverHost string, serverPort int, localPort int) (publicIP string, publicPort int, err error) {
+func natTest(serverHost string, serverPort int, localPort int) (publicIP string, publicPort int, err error) {
-	gLog.Println(LvDEBUG, "natTest start")
+	gLog.Println(LvDEBUG, "natTest start")
-	defer gLog.Println(LvDEBUG, "natTest end")
+	defer gLog.Println(LvDEBUG, "natTest end")
-	conn, err := net.ListenPacket("udp", fmt.Sprintf(":%d", localPort))
+	conn, err := net.ListenPacket("udp", fmt.Sprintf(":%d", localPort))
-	if err != nil {
+	if err != nil {
-		gLog.Println(LvERROR, "natTest listen udp error:", err)
+		gLog.Println(LvERROR, "natTest listen udp error:", err)
-		return "", 0, err
+		return "", 0, err
-	}
+	}
-	defer conn.Close()
+	defer conn.Close()
-
+
-	dst, err := net.ResolveUDPAddr("udp", fmt.Sprintf("%s:%d", serverHost, serverPort))
+	dst, err := net.ResolveUDPAddr("udp", fmt.Sprintf("%s:%d", serverHost, serverPort))
-	if err != nil {
+	if err != nil {
-		return "", 0, err
+		return "", 0, err
-	}
+	}
-
+
-	// The connection can write data to the desired address.
+	// The connection can write data to the desired address.
-	msg, err := newMessage(MsgNATDetect, 0, nil)
+	msg, err := newMessage(MsgNATDetect, 0, nil)
-	_, err = conn.WriteTo(msg, dst)
+	_, err = conn.WriteTo(msg, dst)
-	if err != nil {
+	if err != nil {
-		return "", 0, err
+		return "", 0, err
-	}
+	}
-	deadline := time.Now().Add(NatTestTimeout)
+	deadline := time.Now().Add(NatTestTimeout)
-	err = conn.SetReadDeadline(deadline)
+	err = conn.SetReadDeadline(deadline)
-	if err != nil {
+	if err != nil {
-		return "", 0, err
+		return "", 0, err
-	}
+	}
-	buffer := make([]byte, 1024)
+	buffer := make([]byte, 1024)
-	nRead, _, err := conn.ReadFrom(buffer)
+	nRead, _, err := conn.ReadFrom(buffer)
-	if err != nil {
+	if err != nil {
-		gLog.Println(LvERROR, "NAT detect error:", err)
+		gLog.Println(LvERROR, "NAT detect error:", err)
-		return "", 0, err
+		return "", 0, err
-	}
+	}
-	natRsp := NatDetectRsp{}
+	natRsp := NatDetectRsp{}
-	err = json.Unmarshal(buffer[openP2PHeaderSize:nRead], &natRsp)
+	json.Unmarshal(buffer[openP2PHeaderSize:nRead], &natRsp)
-
+
-	return natRsp.IP, natRsp.Port, nil
+	return natRsp.IP, natRsp.Port, nil
-}
+}
-
+
-func getNATType(host string, udp1 int, udp2 int) (publicIP string, NATType int, hasIPvr int, hasUPNPorNATPMP int, err error) {
+func getNATType(host string, udp1 int, udp2 int) (publicIP string, NATType int, err error) {
-	// the random local port may be used by other.
+	// the random local port may be used by other.
-	localPort := int(rand.Uint32()%15000 + 50000)
+	localPort := int(rand.Uint32()%15000 + 50000)
-	echoPort := P2PNetworkInstance(nil).config.TCPPort
+
-	ip1, port1, err := natTest(host, udp1, localPort)
+	ip1, port1, err := natTest(host, udp1, localPort)
-	if err != nil {
+	if err != nil {
-		return "", 0, 0, 0, err
+		return "", 0, err
-	}
+	}
-	hasIPv4, hasUPNPorNATPMP := publicIPTest(ip1, echoPort)
+	_, port2, err := natTest(host, udp2, localPort) // 2rd nat test not need testing publicip
-	gLog.Printf(LvINFO, "local port:%d, nat port:%d, hasIPv4:%d, UPNP:%d", localPort, port1, hasIPv4, hasUPNPorNATPMP)
+	gLog.Printf(LvDEBUG, "local port:%d  nat port:%d", localPort, port2)
-	_, port2, err := natTest(host, udp2, localPort) // 2rd nat test not need testing publicip
+	if err != nil {
-	gLog.Printf(LvDEBUG, "local port:%d  nat port:%d", localPort, port2)
+		return "", 0, err
-	if err != nil {
+	}
-		return "", 0, hasIPv4, hasUPNPorNATPMP, err
+	natType := NATSymmetric
-	}
+	if port1 == port2 {
-	natType := NATSymmetric
+		natType = NATCone
-	if port1 == port2 {
+	}
-		natType = NATCone
+	return ip1, natType, nil
-	}
+}
-	return ip1, natType, hasIPv4, hasUPNPorNATPMP, nil
+
-}
+func publicIPTest(publicIP string, echoPort int) (hasPublicIP int, hasUPNPorNATPMP int) {
-
+	if publicIP == "" || echoPort == 0 {
-func publicIPTest(publicIP string, echoPort int) (hasPublicIP int, hasUPNPorNATPMP int) {
+		return
-	var echoConn *net.UDPConn
+	}
-	var wg sync.WaitGroup
+	var echoConn *net.UDPConn
-	wg.Add(1)
+	gLog.Println(LvDEBUG, "echo server start")
-	go func() {
+	var err error
-		gLog.Println(LvDEBUG, "echo server start")
+	echoConn, err = net.ListenUDP("udp", &net.UDPAddr{IP: net.IPv4zero, Port: echoPort})
-		var err error
+	if err != nil { // listen error
-		echoConn, err = net.ListenUDP("udp", &net.UDPAddr{IP: net.IPv4zero, Port: echoPort})
+		gLog.Println(LvERROR, "echo server listen error:", err)
-		if err != nil {
+		return
-			gLog.Println(LvERROR, "echo server listen error:", err)
+	}
-			return
+	defer echoConn.Close()
-		}
+	go func() {
-		buf := make([]byte, 1600)
+		// close outside for breaking the ReadFromUDP
-		// close outside for breaking the ReadFromUDP
+		// wait 30s for echo testing
-		// wait 5s for echo testing
+		buf := make([]byte, 1600)
-		wg.Done()
+		echoConn.SetReadDeadline(time.Now().Add(time.Second * 30))
-		echoConn.SetReadDeadline(time.Now().Add(time.Second * 30))
+		n, addr, err := echoConn.ReadFromUDP(buf)
-		n, addr, err := echoConn.ReadFromUDP(buf)
+		if err != nil {
-		if err != nil {
+			return
-			return
+		}
-		}
+		echoConn.WriteToUDP(buf[0:n], addr)
-		echoConn.WriteToUDP(buf[0:n], addr)
+		gLog.Println(LvDEBUG, "echo server end")
-		gLog.Println(LvDEBUG, "echo server end")
+	}()
-	}()
+	// testing for public ip
-	wg.Wait() // wait echo udp
+	for i := 0; i < 2; i++ {
-	defer echoConn.Close()
+		if i == 1 {
-	// testing for public ip
+			// test upnp or nat-pmp
-	for i := 0; i < 2; i++ {
+			gLog.Println(LvDEBUG, "upnp test start")
-		if i == 1 {
+			nat, err := Discover()
-			// test upnp or nat-pmp
+			if err != nil || nat == nil {
-			gLog.Println(LvDEBUG, "upnp test start")
+				gLog.Println(LvDEBUG, "could not perform UPNP discover:", err)
-			nat, err := Discover()
+				break
-			if err != nil || nat == nil {
+			}
-				gLog.Println(LvDEBUG, "could not perform UPNP discover:", err)
+			ext, err := nat.GetExternalAddress()
-				break
+			if err != nil {
-			}
+				gLog.Println(LvDEBUG, "could not perform UPNP external address:", err)
-			ext, err := nat.GetExternalAddress()
+				break
-			if err != nil {
+			}
-				gLog.Println(LvDEBUG, "could not perform UPNP external address:", err)
+			gLog.Println(LvINFO, "PublicIP:", ext)
-				break
+
-			}
+			externalPort, err := nat.AddPortMapping("udp", echoPort, echoPort, "openp2p", 30) // 30 seconds fot upnp testing
-			log.Println("PublicIP:", ext)
+			if err != nil {
-
+				gLog.Println(LvDEBUG, "could not add udp UPNP port mapping", externalPort)
-			externalPort, err := nat.AddPortMapping("udp", echoPort, echoPort, "openp2p", 30)
+				break
-			if err != nil {
+			} else {
-				gLog.Println(LvDEBUG, "could not add udp UPNP port mapping", externalPort)
+				nat.AddPortMapping("tcp", echoPort, echoPort, "openp2p", 604800) // 7 days for tcp connection
-				break
+			}
-			} else {
+		}
-				nat.AddPortMapping("tcp", echoPort, echoPort, "openp2p", 604800)
+		gLog.Printf(LvDEBUG, "public ip test start %s:%d", publicIP, echoPort)
-			}
+		conn, err := net.ListenUDP("udp", nil)
-		}
+		if err != nil {
-		gLog.Printf(LvDEBUG, "public ip test start %s:%d", publicIP, echoPort)
+			break
-		conn, err := net.ListenUDP("udp", nil)
+		}
-		if err != nil {
+		defer conn.Close()
-			break
+		dst, err := net.ResolveUDPAddr("udp", fmt.Sprintf("%s:%d", publicIP, echoPort))
-		}
+		if err != nil {
-		defer conn.Close()
+			break
-		dst, err := net.ResolveUDPAddr("udp", fmt.Sprintf("%s:%d", publicIP, echoPort))
+		}
-		if err != nil {
+		conn.WriteTo([]byte("echo"), dst)
-			break
+		buf := make([]byte, 1600)
-		}
+
-		conn.WriteTo([]byte("echo"), dst)
+		// wait for echo testing
-		buf := make([]byte, 1600)
+		conn.SetReadDeadline(time.Now().Add(PublicIPEchoTimeout))
-
+		_, _, err = conn.ReadFromUDP(buf)
-		// wait for echo testing
+		if err == nil {
-		conn.SetReadDeadline(time.Now().Add(PublicIPEchoTimeout))
+			if i == 1 {
-		_, _, err = conn.ReadFromUDP(buf)
+				gLog.Println(LvDEBUG, "UPNP or NAT-PMP:YES")
-		if err == nil {
+				hasUPNPorNATPMP = 1
-			if i == 1 {
+			} else {
-				gLog.Println(LvDEBUG, "UPNP or NAT-PMP:YES")
+				gLog.Println(LvDEBUG, "public ip:YES")
-				hasUPNPorNATPMP = 1
+				hasPublicIP = 1
-			} else {
+			}
-				gLog.Println(LvDEBUG, "public ip:YES")
+			break
-				hasPublicIP = 1
+		}
-			}
+	}
-			break
+	return
-		}
+}
-	}
-	return
-}

core/openp2p.go

@@ -1,100 +1,121 @@
-package openp2p
+package openp2p
-
+
-import (
+import (
-	"fmt"
+	"fmt"
-	"math/rand"
+	"math/rand"
-	"os"
+	"os"
-	"path/filepath"
+	"path/filepath"
-	"strconv"
+	"strconv"
-	"time"
+	"time"
-)
+)
-
+
-func Run() {
+var GNetwork *P2PNetwork
-	rand.Seed(time.Now().UnixNano())
+
-	baseDir := filepath.Dir(os.Args[0])
+func Run() {
-	os.Chdir(baseDir) // for system service
+	rand.Seed(time.Now().UnixNano())
-	gLog = NewLogger(baseDir, ProductName, LvDEBUG, 1024*1024, LogFileAndConsole)
+	baseDir := filepath.Dir(os.Args[0])
-	// TODO: install sub command, deamon process
+	os.Chdir(baseDir) // for system service
-	if len(os.Args) > 1 {
+	gLog = NewLogger(baseDir, ProductName, LvDEBUG, 1024*1024, LogFile|LogConsole)
-		switch os.Args[1] {
+	if len(os.Args) > 1 {
-		case "version", "-v", "--version":
+		switch os.Args[1] {
-			fmt.Println(OpenP2PVersion)
+		case "version", "-v", "--version":
-			return
+			fmt.Println(OpenP2PVersion)
-		case "update":
+			return
-			gLog = NewLogger(baseDir, ProductName, LvDEBUG, 1024*1024, LogFileAndConsole)
+		case "install":
-			targetPath := filepath.Join(defaultInstallPath, defaultBinName)
+			install()
-			d := daemon{}
+			return
-			err := d.Control("restart", targetPath, nil)
+		case "uninstall":
-			if err != nil {
+			uninstall()
-				gLog.Println(LvERROR, "restart service error:", err)
+			return
-			} else {
+		}
-				gLog.Println(LvINFO, "restart service ok.")
+	} else {
-			}
+		installByFilename()
-			return
+	}
-		case "install":
+	parseParams("", "")
-			install()
+	gLog.Println(LvINFO, "openp2p start. version: ", OpenP2PVersion)
-			return
+	gLog.Println(LvINFO, "Contact: QQ group 16947733, Email [email protected]")
-		case "uninstall":
+
-			uninstall()
+	if gConf.daemonMode {
-			return
+		d := daemon{}
-		}
+		d.run()
-	} else {
+		return
-		installByFilename()
+	}
-	}
+
-	parseParams("")
+	gLog.Println(LvINFO, &gConf)
-	gLog.Println(LvINFO, "openp2p start. version: ", OpenP2PVersion)
+	setFirewall()
-	gLog.Println(LvINFO, "Contact: QQ group 16947733, Email [email protected]")
+	err := setRLimit()
-
+	if err != nil {
-	if gConf.daemonMode {
+		gLog.Println(LvINFO, "setRLimit error:", err)
-		d := daemon{}
+	}
-		d.run()
+	GNetwork = P2PNetworkInstance()
-		return
+	if ok := GNetwork.Connect(30000); !ok {
-	}
+		gLog.Println(LvERROR, "P2PNetwork login error")
-
+		return
-	gLog.Println(LvINFO, &gConf)
+	}
-	setFirewall()
+	// gLog.Println(LvINFO, "waiting for connection...")
-	network := P2PNetworkInstance(&gConf.Network)
+	forever := make(chan bool)
-	if ok := network.Connect(30000); !ok {
+	<-forever
-		gLog.Println(LvERROR, "P2PNetwork login error")
+}
-		return
+
-	}
+// for Android app
-	gLog.Println(LvINFO, "waiting for connection...")
+// gomobile not support uint64 exported to java
-	forever := make(chan bool)
+
-	<-forever
+func RunAsModule(baseDir string, token string, bw int, logLevel int) *P2PNetwork {
-}
+	rand.Seed(time.Now().UnixNano())
-
+	os.Chdir(baseDir) // for system service
-var network *P2PNetwork
+	gLog = NewLogger(baseDir, ProductName, LvINFO, 1024*1024, LogFile|LogConsole)
-
+
-// for Android app
+	parseParams("", "")
-// gomobile not support uint64 exported to java
+
-func RunAsModule(baseDir string, token string, bw int, logLevel int) *P2PNetwork {
+	n, err := strconv.ParseUint(token, 10, 64)
-	rand.Seed(time.Now().UnixNano())
+	if err == nil && n > 0 {
-	os.Chdir(baseDir) // for system service
+		gConf.setToken(n)
-	gLog = NewLogger(baseDir, ProductName, LvDEBUG, 1024*1024, LogFileAndConsole)
+	}
-
+	if n <= 0 && gConf.Network.Token == 0 { // not input token
-	parseParams("")
+		return nil
-
+	}
-	n, err := strconv.ParseUint(token, 10, 64)
+	// gLog.setLevel(LogLevel(logLevel))
-	if err == nil {
+	gConf.setShareBandwidth(bw)
-		gConf.setToken(n)
+	gLog.Println(LvINFO, "openp2p start. version: ", OpenP2PVersion)
-	}
+	gLog.Println(LvINFO, "Contact: QQ group 16947733, Email [email protected]")
-	gLog.setLevel(LogLevel(logLevel))
+	gLog.Println(LvINFO, &gConf)
-	gConf.setShareBandwidth(bw)
+
-	gLog.Println(LvINFO, "openp2p start. version: ", OpenP2PVersion)
+	GNetwork = P2PNetworkInstance()
-	gLog.Println(LvINFO, "Contact: QQ group 16947733, Email [email protected]")
+	if ok := GNetwork.Connect(30000); !ok {
-	gLog.Println(LvINFO, &gConf)
+		gLog.Println(LvERROR, "P2PNetwork login error")
-
+		return nil
-	network = P2PNetworkInstance(&gConf.Network)
+	}
-	if ok := network.Connect(30000); !ok {
+	// gLog.Println(LvINFO, "waiting for connection...")
-		gLog.Println(LvERROR, "P2PNetwork login error")
+	return GNetwork
-		return nil
+}
-	}
+
-	gLog.Println(LvINFO, "waiting for connection...")
+func RunCmd(cmd string) {
-	return network
+	rand.Seed(time.Now().UnixNano())
-}
+	baseDir := filepath.Dir(os.Args[0])
-
+	os.Chdir(baseDir) // for system service
-func GetToken(baseDir string) string {
+	gLog = NewLogger(baseDir, ProductName, LvINFO, 1024*1024, LogFile|LogConsole)
-	os.Chdir(baseDir)
+
-	gConf.load()
+	parseParams("", cmd)
-	return fmt.Sprintf("%d", gConf.Network.Token)
+	setFirewall()
-}
+	err := setRLimit()
+	if err != nil {
+		gLog.Println(LvINFO, "setRLimit error:", err)
+	}
+	GNetwork = P2PNetworkInstance()
+	if ok := GNetwork.Connect(30000); !ok {
+		gLog.Println(LvERROR, "P2PNetwork login error")
+		return
+	}
+	forever := make(chan bool)
+	<-forever
+}
+
+func GetToken(baseDir string) string {
+	os.Chdir(baseDir)
+	gConf.load()
+	return fmt.Sprintf("%d", gConf.Network.Token)
+}
+
+func Stop() {
+	os.Exit(0)
+}

core/optun.go

@@ -0,0 +1,20 @@
+package openp2p
+
+import (
+	"github.com/openp2p-cn/wireguard-go/tun"
+)
+
+var AndroidSDWANConfig chan []byte
+
+type optun struct {
+	tunName string
+	dev     tun.Device
+}
+
+func (t *optun) Stop() error {
+	t.dev.Close()
+	return nil
+}
+func init() {
+	AndroidSDWANConfig = make(chan []byte, 1)
+}

core/optun_android.go

@@ -0,0 +1,85 @@
+// optun_android.go
+//go:build android
+// +build android
+
+package openp2p
+
+import (
+	"net"
+)
+
+const (
+	tunIfaceName = "optun"
+	PIHeaderSize = 0
+)
+
+var AndroidReadTun chan []byte // TODO: multi channel
+var AndroidWriteTun chan []byte
+
+func (t *optun) Start(localAddr string, detail *SDWANInfo) error {
+
+	return nil
+}
+
+func (t *optun) Read(bufs [][]byte, sizes []int, offset int) (n int, err error) {
+	bufs[0] = <-AndroidReadTun
+	sizes[0] = len(bufs[0])
+	return 1, nil
+}
+
+func (t *optun) Write(bufs [][]byte, offset int) (int, error) {
+	AndroidWriteTun <- bufs[0]
+	return len(bufs[0]), nil
+}
+
+func AndroidRead(data []byte, len int) {
+	head := PacketHeader{}
+	parseHeader(data, &head)
+	gLog.Printf(LvDev, "AndroidRead tun dst ip=%s,len=%d", net.IP{byte(head.dst >> 24), byte(head.dst >> 16), byte(head.dst >> 8), byte(head.dst)}.String(), len)
+	buf := make([]byte, len)
+	copy(buf, data)
+	AndroidReadTun <- buf
+}
+
+func AndroidWrite(buf []byte) int {
+	p := <-AndroidWriteTun
+	copy(buf, p)
+	return len(p)
+}
+
+func GetAndroidSDWANConfig(buf []byte) int {
+	p := <-AndroidSDWANConfig
+	copy(buf, p)
+	gLog.Printf(LvINFO, "AndroidSDWANConfig=%s", p)
+	return len(p)
+}
+
+func GetAndroidNodeName() string {
+	gLog.Printf(LvINFO, "GetAndroidNodeName=%s", gConf.Network.Node)
+	return gConf.Network.Node
+}
+
+func setTunAddr(ifname, localAddr, remoteAddr string, wintun interface{}) error {
+	// TODO:
+	return nil
+}
+
+func addRoute(dst, gw, ifname string) error {
+	// TODO:
+	return nil
+}
+
+func delRoute(dst, gw string) error {
+	// TODO:
+	return nil
+}
+
+func delRoutesByGateway(gateway string) error {
+	// TODO:
+	return nil
+}
+
+func init() {
+	AndroidReadTun = make(chan []byte, 1000)
+	AndroidWriteTun = make(chan []byte, 1000)
+}

core/optun_darwin.go

@@ -0,0 +1,87 @@
+package openp2p
+
+import (
+	"fmt"
+	"net"
+	"os/exec"
+	"strings"
+
+	"github.com/openp2p-cn/wireguard-go/tun"
+)
+
+const (
+	tunIfaceName = "utun"
+	PIHeaderSize = 4 // utun has no IFF_NO_PI
+)
+
+func (t *optun) Start(localAddr string, detail *SDWANInfo) error {
+	var err error
+	t.tunName = tunIfaceName
+	t.dev, err = tun.CreateTUN(t.tunName, 1420)
+	if err != nil {
+		return err
+	}
+	t.tunName, _ = t.dev.Name()
+	return nil
+}
+
+func (t *optun) Read(bufs [][]byte, sizes []int, offset int) (n int, err error) {
+	return t.dev.Read(bufs, sizes, offset)
+}
+
+func (t *optun) Write(bufs [][]byte, offset int) (int, error) {
+	return t.dev.Write(bufs, offset)
+}
+
+func setTunAddr(ifname, localAddr, remoteAddr string, wintun interface{}) error {
+	li, _, err := net.ParseCIDR(localAddr)
+	if err != nil {
+		return fmt.Errorf("parse local addr fail:%s", err)
+	}
+	ri, _, err := net.ParseCIDR(remoteAddr)
+	if err != nil {
+		return fmt.Errorf("parse remote addr fail:%s", err)
+	}
+	err = exec.Command("ifconfig", ifname, "inet", li.String(), ri.String(), "up").Run()
+	return err
+}
+
+func addRoute(dst, gw, ifname string) error {
+	err := exec.Command("route", "add", dst, gw).Run()
+	return err
+}
+
+func delRoute(dst, gw string) error {
+	err := exec.Command("route", "delete", dst, gw).Run()
+	return err
+}
+func delRoutesByGateway(gateway string) error {
+	cmd := exec.Command("netstat", "-rn")
+	output, err := cmd.Output()
+	if err != nil {
+		return err
+	}
+
+	lines := strings.Split(string(output), "\n")
+	for _, line := range lines {
+		if !strings.Contains(line, gateway) {
+			continue
+		}
+		fields := strings.Fields(line)
+		if len(fields) >= 7 && fields[0] == "default" && fields[len(fields)-1] == gateway {
+			delCmd := exec.Command("route", "delete", "default", gateway)
+			err := delCmd.Run()
+			if err != nil {
+				return err
+			}
+			fmt.Printf("Delete route ok: %s %s\n", "default", gateway)
+		}
+	}
+	return nil
+}
+func addTunAddr(localAddr, remoteAddr string) error {
+	return nil
+}
+func delTunAddr(localAddr, remoteAddr string) error {
+	return nil
+}

core/optun_linux.go

@@ -0,0 +1,133 @@
+//go:build !android
+// +build !android
+
+// optun_linux.go
+package openp2p
+
+import (
+	"fmt"
+	"net"
+	"os/exec"
+	"strings"
+
+	"github.com/openp2p-cn/wireguard-go/tun"
+	"github.com/vishvananda/netlink"
+)
+
+const (
+	tunIfaceName = "optun"
+	PIHeaderSize = 0
+)
+
+var previousIP = ""
+
+func (t *optun) Start(localAddr string, detail *SDWANInfo) error {
+	var err error
+	t.tunName = tunIfaceName
+	t.dev, err = tun.CreateTUN(t.tunName, 1420)
+	if err != nil {
+		return err
+	}
+	return nil
+}
+
+func (t *optun) Read(bufs [][]byte, sizes []int, offset int) (n int, err error) {
+	return t.dev.Read(bufs, sizes, offset)
+}
+
+func (t *optun) Write(bufs [][]byte, offset int) (int, error) {
+	return t.dev.Write(bufs, offset)
+}
+
+func setTunAddr(ifname, localAddr, remoteAddr string, wintun interface{}) error {
+	ifce, err := netlink.LinkByName(ifname)
+	if err != nil {
+		return err
+	}
+	netlink.LinkSetMTU(ifce, 1375)
+	netlink.LinkSetTxQLen(ifce, 100)
+	netlink.LinkSetUp(ifce)
+
+	ln, err := netlink.ParseIPNet(localAddr)
+	if err != nil {
+		return err
+	}
+	ln.Mask = net.CIDRMask(32, 32)
+	rn, err := netlink.ParseIPNet(remoteAddr)
+	if err != nil {
+		return err
+	}
+	rn.Mask = net.CIDRMask(32, 32)
+
+	addr := &netlink.Addr{
+		IPNet: ln,
+		Peer:  rn,
+	}
+	if previousIP != "" {
+		lnDel, err := netlink.ParseIPNet(previousIP)
+		if err != nil {
+			return err
+		}
+		lnDel.Mask = net.CIDRMask(32, 32)
+
+		addrDel := &netlink.Addr{
+			IPNet: lnDel,
+			Peer:  rn,
+		}
+		netlink.AddrDel(ifce, addrDel)
+	}
+	previousIP = localAddr
+	return netlink.AddrAdd(ifce, addr)
+}
+
+func addRoute(dst, gw, ifname string) error {
+	_, networkid, err := net.ParseCIDR(dst)
+	if err != nil {
+		return err
+	}
+	ipGW := net.ParseIP(gw)
+	if ipGW == nil {
+		return fmt.Errorf("parse gateway %s failed", gw)
+	}
+	route := &netlink.Route{
+		Dst: networkid,
+		Gw:  ipGW,
+	}
+	return netlink.RouteAdd(route)
+}
+
+func delRoute(dst, gw string) error {
+	_, networkid, err := net.ParseCIDR(dst)
+	if err != nil {
+		return err
+	}
+	route := &netlink.Route{
+		Dst: networkid,
+	}
+	return netlink.RouteDel(route)
+}
+
+func delRoutesByGateway(gateway string) error {
+	cmd := exec.Command("route", "-n")
+	output, err := cmd.Output()
+	if err != nil {
+		return err
+	}
+
+	lines := strings.Split(string(output), "\n")
+	for _, line := range lines {
+		if !strings.Contains(line, gateway) {
+			continue
+		}
+		fields := strings.Fields(line)
+		if len(fields) >= 8 && fields[1] == "0.0.0.0" && fields[7] == gateway {
+			delCmd := exec.Command("route", "del", "-net", fields[0], "gw", gateway)
+			err := delCmd.Run()
+			if err != nil {
+				return err
+			}
+			fmt.Printf("Delete route ok: %s %s %s\n", fields[0], fields[1], gateway)
+		}
+	}
+	return nil
+}

core/optun_other.go

@@ -0,0 +1,42 @@
+//go:build !linux && !windows && !darwin
+// +build !linux,!windows,!darwin
+
+package openp2p
+
+import "github.com/openp2p-cn/wireguard-go/tun"
+
+const (
+	tunIfaceName = "optun"
+	PIHeaderSize = 0
+)
+
+func (t *optun) Start(localAddr string, detail *SDWANInfo) error {
+	var err error
+	t.tunName = tunIfaceName
+	t.dev, err = tun.CreateTUN(t.tunName, 1420)
+
+	if err != nil {
+		return err
+	}
+	err = setTunAddr(t.tunName, localAddr, detail.Gateway, t.dev)
+	if err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func addRoute(dst, gw, ifname string) error {
+	return nil
+}
+
+func delRoute(dst, gw string) error {
+	return nil
+}
+func addTunAddr(localAddr, remoteAddr string) error {
+	return nil
+}
+
+func delTunAddr(localAddr, remoteAddr string) error {
+	return nil
+}

core/optun_windows.go

@@ -0,0 +1,142 @@
+package openp2p
+
+import (
+	"fmt"
+	"net"
+	"net/netip"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"runtime"
+	"strconv"
+	"strings"
+
+	"github.com/openp2p-cn/wireguard-go/tun"
+	"golang.org/x/sys/windows"
+	"golang.zx2c4.com/wireguard/windows/tunnel/winipcfg"
+)
+
+const (
+	tunIfaceName = "optun"
+	PIHeaderSize = 0
+)
+
+func (t *optun) Start(localAddr string, detail *SDWANInfo) error {
+	// check wintun.dll
+	tmpFile := filepath.Dir(os.Args[0]) + "/wintun.dll"
+	fs, err := os.Stat(tmpFile)
+	if err != nil || fs.Size() == 0 {
+		url := fmt.Sprintf("https://openp2p.cn/download/v1/latest/wintun/%s/wintun.dll", runtime.GOARCH)
+		err = downloadFile(url, "", tmpFile)
+		if err != nil {
+			os.Remove(tmpFile)
+			return err
+		}
+	}
+
+	t.tunName = tunIfaceName
+
+	uuid := &windows.GUID{
+		Data1: 0xf411e821,
+		Data2: 0xb310,
+		Data3: 0x4567,
+		Data4: [8]byte{0x80, 0x42, 0x83, 0x7e, 0xf4, 0x56, 0xce, 0x13},
+	}
+	t.dev, err = tun.CreateTUNWithRequestedGUID(t.tunName, uuid, 1420)
+	if err != nil { // retry
+		t.dev, err = tun.CreateTUNWithRequestedGUID(t.tunName, uuid, 1420)
+	}
+
+	if err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func (t *optun) Read(bufs [][]byte, sizes []int, offset int) (n int, err error) {
+	return t.dev.Read(bufs, sizes, offset)
+}
+
+func (t *optun) Write(bufs [][]byte, offset int) (int, error) {
+	return t.dev.Write(bufs, offset)
+}
+
+func setTunAddr(ifname, localAddr, remoteAddr string, wintun interface{}) error {
+	nativeTunDevice := wintun.(*tun.NativeTun)
+	link := winipcfg.LUID(nativeTunDevice.LUID())
+	ip, err := netip.ParsePrefix(localAddr)
+	if err != nil {
+		gLog.Printf(LvERROR, "ParsePrefix error:%s, luid:%d,localAddr:%s", err, nativeTunDevice.LUID(), localAddr)
+		return err
+	}
+	err = link.SetIPAddresses([]netip.Prefix{ip})
+	if err != nil {
+		gLog.Printf(LvERROR, "SetIPAddresses error:%s, netip.Prefix:%+v", err, []netip.Prefix{ip})
+		return err
+	}
+	return nil
+}
+
+func addRoute(dst, gw, ifname string) error {
+	_, dstNet, err := net.ParseCIDR(dst)
+	if err != nil {
+		return err
+	}
+	i, err := net.InterfaceByName(ifname)
+	if err != nil {
+		return err
+	}
+	params := make([]string, 0)
+	params = append(params, "add")
+	params = append(params, dstNet.IP.String())
+	params = append(params, "mask")
+	params = append(params, net.IP(dstNet.Mask).String())
+	params = append(params, gw)
+	params = append(params, "if")
+	params = append(params, strconv.Itoa(i.Index))
+	// gLogger.Println(LevelINFO, "windows add route params:", params)
+	execCommand("route", true, params...)
+	return nil
+}
+
+func delRoute(dst, gw string) error {
+	_, dstNet, err := net.ParseCIDR(dst)
+	if err != nil {
+		return err
+	}
+	params := make([]string, 0)
+	params = append(params, "delete")
+	params = append(params, dstNet.IP.String())
+	params = append(params, "mask")
+	params = append(params, net.IP(dstNet.Mask).String())
+	params = append(params, gw)
+	// gLogger.Println(LevelINFO, "windows delete route params:", params)
+	execCommand("route", true, params...)
+	return nil
+}
+
+func delRoutesByGateway(gateway string) error {
+	cmd := exec.Command("route", "print", "-4")
+	output, err := cmd.Output()
+	if err != nil {
+		return err
+	}
+
+	lines := strings.Split(string(output), "\n")
+	for _, line := range lines {
+		if !strings.Contains(line, gateway) {
+			continue
+		}
+		fields := strings.Fields(line)
+		if len(fields) >= 5 {
+			cmd := exec.Command("route", "delete", fields[0], "mask", fields[1], gateway)
+			err := cmd.Run()
+			if err != nil {
+				fmt.Println("Delete route error:", err)
+			}
+			fmt.Printf("Delete route ok: %s %s %s\n", fields[0], fields[1], gateway)
+		}
+	}
+	return nil
+}

core/overlay.go

@@ -23,36 +23,36 @@ func (e *DeadlineExceededError) Temporary() bool { return true }
 
 // implement io.Writer
 type overlayConn struct {
-	tunnel      *P2PTunnel
+	tunnel      *P2PTunnel // TODO: del
+	app         *p2pApp
 	connTCP     net.Conn
 	id          uint64
 	rtid        uint64
 	running     bool
 	isClient    bool
-	appID       uint64
+	appID       uint64 // TODO: del
-	appKey      uint64
+	appKey      uint64 // TODO: del
-	appKeyBytes []byte
+	appKeyBytes []byte // TODO: del
 	// for udp
 	connUDP       *net.UDPConn
 	remoteAddr    net.Addr
-	udpRelayData  chan []byte
+	udpData       chan []byte
 	lastReadUDPTs time.Time
 }
 
 func (oConn *overlayConn) run() {
 	gLog.Printf(LvDEBUG, "%d overlayConn run start", oConn.id)
 	defer gLog.Printf(LvDEBUG, "%d overlayConn run end", oConn.id)
-	oConn.running = true
 	oConn.lastReadUDPTs = time.Now()
-	buffer := make([]byte, ReadBuffLen+PaddingSize)
+	buffer := make([]byte, ReadBuffLen+PaddingSize) // 16 bytes for padding
-	readBuf := buffer[:ReadBuffLen]
+	reuseBuff := buffer[:ReadBuffLen]
 	encryptData := make([]byte, ReadBuffLen+PaddingSize) // 16 bytes for padding
 	tunnelHead := new(bytes.Buffer)
 	relayHead := new(bytes.Buffer)
 	binary.Write(relayHead, binary.LittleEndian, oConn.rtid)
 	binary.Write(tunnelHead, binary.LittleEndian, oConn.id)
 	for oConn.running && oConn.tunnel.isRuning() {
-		buff, dataLen, err := oConn.Read(readBuf)
+		readBuff, dataLen, err := oConn.Read(reuseBuff)
 		if err != nil {
 			if ne, ok := err.(net.Error); ok && ne.Timeout() {
 				continue
@@ -61,20 +61,21 @@ func (oConn *overlayConn) run() {
 			gLog.Printf(LvDEBUG, "overlayConn %d read error:%s,close it", oConn.id, err)
 			break
 		}
-		payload := buff[:dataLen]
+		payload := readBuff[:dataLen]
 		if oConn.appKey != 0 {
-			payload, _ = encryptBytes(oConn.appKeyBytes, encryptData, buffer[:dataLen], dataLen)
+			payload, _ = encryptBytes(oConn.appKeyBytes, encryptData, readBuff[:dataLen], dataLen)
 		}
 		writeBytes := append(tunnelHead.Bytes(), payload...)
+		// TODO: app.write
 		if oConn.rtid == 0 {
 			oConn.tunnel.conn.WriteBytes(MsgP2P, MsgOverlayData, writeBytes)
-			gLog.Printf(LvDEBUG, "write overlay data to %d:%d bodylen=%d", oConn.rtid, oConn.id, len(writeBytes))
+			gLog.Printf(LvDev, "write overlay data to tid:%d,oid:%d bodylen=%d", oConn.tunnel.id, oConn.id, len(writeBytes))
 		} else {
 			// write raley data
 			all := append(relayHead.Bytes(), encodeHeader(MsgP2P, MsgOverlayData, uint32(len(writeBytes)))...)
 			all = append(all, writeBytes...)
 			oConn.tunnel.conn.WriteBytes(MsgP2P, MsgRelayData, all)
-			gLog.Printf(LvDEBUG, "write relay data to %d:%d bodylen=%d", oConn.rtid, oConn.id, len(writeBytes))
+			gLog.Printf(LvDev, "write relay data to tid:%d,rtid:%d,oid:%d bodylen=%d", oConn.tunnel.id, oConn.rtid, oConn.id, len(writeBytes))
 		}
 	}
 	if oConn.connTCP != nil {
@@ -85,20 +86,15 @@ func (oConn *overlayConn) run() {
 	}
 	oConn.tunnel.overlayConns.Delete(oConn.id)
 	// notify peer disconnect
-	if oConn.isClient {
+	req := OverlayDisconnectReq{ID: oConn.id}
-		req := OverlayDisconnectReq{ID: oConn.id}
+	oConn.tunnel.WriteMessage(oConn.rtid, MsgP2P, MsgOverlayDisconnectReq, &req)
-		if oConn.rtid == 0 {
-			oConn.tunnel.conn.WriteMessage(MsgP2P, MsgOverlayDisconnectReq, &req)
-		} else {
-			// write relay data
-			msg, _ := newMessage(MsgP2P, MsgOverlayDisconnectReq, &req)
-			msgWithHead := append(relayHead.Bytes(), msg...)
-			oConn.tunnel.conn.WriteBytes(MsgP2P, MsgRelayData, msgWithHead)
-		}
-	}
 }
 
-func (oConn *overlayConn) Read(reuseBuff []byte) (buff []byte, n int, err error) {
+func (oConn *overlayConn) Read(reuseBuff []byte) (buff []byte, dataLen int, err error) {
+	if !oConn.running {
+		err = ErrOverlayConnDisconnect
+		return
+	}
 	if oConn.connUDP != nil {
 		if time.Now().After(oConn.lastReadUDPTs.Add(time.Minute * 5)) {
 			err = errors.New("udp close")
@@ -106,15 +102,15 @@ func (oConn *overlayConn) Read(reuseBuff []byte) (buff []byte, n int, err error)
 		}
 		if oConn.remoteAddr != nil { // as server
 			select {
-			case buff = <-oConn.udpRelayData:
+			case buff = <-oConn.udpData:
-				n = len(buff)
+				dataLen = len(buff) - PaddingSize
 				oConn.lastReadUDPTs = time.Now()
 			case <-time.After(time.Second * 10):
 				err = ErrDeadlineExceeded
 			}
 		} else { // as client
-			oConn.connUDP.SetReadDeadline(time.Now().Add(5 * time.Second))
+			oConn.connUDP.SetReadDeadline(time.Now().Add(UDPReadTimeout))
-			n, _, err = oConn.connUDP.ReadFrom(reuseBuff)
+			dataLen, _, err = oConn.connUDP.ReadFrom(reuseBuff)
 			if err == nil {
 				oConn.lastReadUDPTs = time.Now()
 			}
@@ -122,15 +118,21 @@ func (oConn *overlayConn) Read(reuseBuff []byte) (buff []byte, n int, err error)
 		}
 		return
 	}
-	oConn.connTCP.SetReadDeadline(time.Now().Add(time.Second * 5))
+	if oConn.connTCP != nil {
-	n, err = oConn.connTCP.Read(reuseBuff)
+		oConn.connTCP.SetReadDeadline(time.Now().Add(UDPReadTimeout))
-	buff = reuseBuff
+		dataLen, err = oConn.connTCP.Read(reuseBuff)
+		buff = reuseBuff
+	}
+
 	return
 }
 
 // calling by p2pTunnel
 func (oConn *overlayConn) Write(buff []byte) (n int, err error) {
 	// add mutex when multi-thread calling
+	if !oConn.running {
+		return 0, ErrOverlayConnDisconnect
+	}
 	if oConn.connUDP != nil {
 		if oConn.remoteAddr == nil {
 			n, err = oConn.connUDP.Write(buff)
@@ -142,9 +144,25 @@ func (oConn *overlayConn) Write(buff []byte) (n int, err error) {
 		}
 		return
 	}
-	n, err = oConn.connTCP.Write(buff)
+	if oConn.connTCP != nil {
+		n, err = oConn.connTCP.Write(buff)
+	}
+
 	if err != nil {
 		oConn.running = false
 	}
 	return
 }
+
+func (oConn *overlayConn) Close() (err error) {
+	oConn.running = false
+	if oConn.connTCP != nil {
+		oConn.connTCP.Close()
+		// oConn.connTCP = nil
+	}
+	if oConn.connUDP != nil {
+		oConn.connUDP.Close()
+		// oConn.connUDP = nil
+	}
+	return nil
+}

core/p2papp.go

@@ -13,49 +13,386 @@ import (
 )
 
 type p2pApp struct {
-	config      AppConfig
+	config       AppConfig
-	listener    net.Listener
+	listener     net.Listener
-	listenerUDP *net.UDPConn
+	listenerUDP  *net.UDPConn
-	tunnel      *P2PTunnel
+	directTunnel *P2PTunnel
-	rtid        uint64 // relay tunnelID
+	relayTunnel  *P2PTunnel
-	relayNode   string
+	tunnelMtx    sync.Mutex
-	relayMode   string
+	iptree       *IPTree // for whitelist
-	hbTime      time.Time
+	rtid         uint64  // relay tunnelID
-	hbMtx       sync.Mutex
+	relayNode    string
-	running     bool
+	relayMode    string // public/private
-	id          uint64
+	hbTimeRelay  time.Time
-	key         uint64
+	hbMtx        sync.Mutex
-	wg          sync.WaitGroup
+	running      bool
+	id           uint64
+	key          uint64 // aes
+	wg           sync.WaitGroup
+	relayHead    *bytes.Buffer
+	once         sync.Once
+	// for relayTunnel
+	retryRelayNum      int
+	retryRelayTime     time.Time
+	nextRetryRelayTime time.Time
+	errMsg             string
+	connectTime        time.Time
+}
+
+func (app *p2pApp) Tunnel() *P2PTunnel {
+	app.tunnelMtx.Lock()
+	defer app.tunnelMtx.Unlock()
+	if app.directTunnel != nil {
+		return app.directTunnel
+	}
+	return app.relayTunnel
+}
+
+func (app *p2pApp) DirectTunnel() *P2PTunnel {
+	app.tunnelMtx.Lock()
+	defer app.tunnelMtx.Unlock()
+	return app.directTunnel
+}
+
+func (app *p2pApp) setDirectTunnel(t *P2PTunnel) {
+	app.tunnelMtx.Lock()
+	defer app.tunnelMtx.Unlock()
+	app.directTunnel = t
+}
+
+func (app *p2pApp) RelayTunnel() *P2PTunnel {
+	app.tunnelMtx.Lock()
+	defer app.tunnelMtx.Unlock()
+	return app.relayTunnel
+}
+
+func (app *p2pApp) setRelayTunnel(t *P2PTunnel) {
+	app.tunnelMtx.Lock()
+	defer app.tunnelMtx.Unlock()
+	app.relayTunnel = t
+}
+
+func (app *p2pApp) isDirect() bool {
+	return app.directTunnel != nil
+}
+
+func (app *p2pApp) RelayTunnelID() uint64 {
+	if app.isDirect() {
+		return 0
+	}
+	return app.rtid
+}
+
+func (app *p2pApp) ConnectTime() time.Time {
+	if app.isDirect() {
+		return app.config.connectTime
+	}
+	return app.connectTime
+}
+
+func (app *p2pApp) RetryTime() time.Time {
+	if app.isDirect() {
+		return app.config.retryTime
+	}
+	return app.retryRelayTime
+}
+
+func (app *p2pApp) checkP2PTunnel() error {
+	for app.running {
+		app.checkDirectTunnel()
+		app.checkRelayTunnel()
+		time.Sleep(time.Second * 3)
+	}
+	return nil
+}
+
+func (app *p2pApp) directRetryLimit() int {
+	if app.config.peerIP == gConf.Network.publicIP && compareVersion(app.config.peerVersion, SupportIntranetVersion) >= 0 {
+		return retryLimit
+	}
+	if IsIPv6(app.config.peerIPv6) && IsIPv6(gConf.IPv6()) {
+		return retryLimit
+	}
+	if app.config.hasIPv4 == 1 || gConf.Network.hasIPv4 == 1 || app.config.hasUPNPorNATPMP == 1 || gConf.Network.hasUPNPorNATPMP == 1 {
+		return retryLimit
+	}
+	if gConf.Network.natType == NATCone && app.config.peerNatType == NATCone {
+		return retryLimit
+	}
+	if app.config.peerNatType == NATSymmetric && gConf.Network.natType == NATSymmetric {
+		return 0
+	}
+	return retryLimit / 10 // c2s or s2c
+}
+func (app *p2pApp) checkDirectTunnel() error {
+	if app.config.ForceRelay == 1 && app.config.RelayNode != app.config.PeerNode {
+		return nil
+	}
+	if app.DirectTunnel() != nil && app.DirectTunnel().isActive() {
+		return nil
+	}
+	if app.config.nextRetryTime.After(time.Now()) || app.config.Enabled == 0 || app.config.retryNum >= app.directRetryLimit() {
+		return nil
+	}
+	if time.Now().Add(-time.Minute * 15).After(app.config.retryTime) { // run normally 15min, reset retrynum
+		app.config.retryNum = 1
+	}
+	if app.config.retryNum > 0 { // first time not show reconnect log
+		gLog.Printf(LvINFO, "detect app %s appid:%d disconnect, reconnecting the %d times...", app.config.LogPeerNode(), app.id, app.config.retryNum)
+	}
+	app.config.retryNum++
+	app.config.retryTime = time.Now()
+	app.config.nextRetryTime = time.Now().Add(retryInterval)
+	app.config.connectTime = time.Now()
+	err := app.buildDirectTunnel()
+	if err != nil {
+		app.config.errMsg = err.Error()
+		if err == ErrPeerOffline && app.config.retryNum > 2 { // stop retry, waiting for online
+			app.config.retryNum = retryLimit
+			gLog.Printf(LvINFO, " %s offline, it will auto reconnect when peer node online", app.config.LogPeerNode())
+		}
+		if err == ErrBuildTunnelBusy {
+			app.config.retryNum--
+		}
+	}
+	if app.Tunnel() != nil {
+		app.once.Do(func() {
+			go app.listen()
+			// memapp also need
+			go app.relayHeartbeatLoop()
+		})
+	}
+	return nil
+}
+func (app *p2pApp) buildDirectTunnel() error {
+	relayNode := ""
+	peerNatType := NATUnknown
+	peerIP := ""
+	errMsg := ""
+	var t *P2PTunnel
+	var err error
+	pn := GNetwork
+	initErr := pn.requestPeerInfo(&app.config)
+	if initErr != nil {
+		gLog.Printf(LvERROR, "%s init error:%s", app.config.LogPeerNode(), initErr)
+		return initErr
+	}
+	t, err = pn.addDirectTunnel(app.config, 0)
+	if t != nil {
+		peerNatType = t.config.peerNatType
+		peerIP = t.config.peerIP
+	}
+	if err != nil {
+		errMsg = err.Error()
+	}
+	req := ReportConnect{
+		Error:          errMsg,
+		Protocol:       app.config.Protocol,
+		SrcPort:        app.config.SrcPort,
+		NatType:        gConf.Network.natType,
+		PeerNode:       app.config.PeerNode,
+		DstPort:        app.config.DstPort,
+		DstHost:        app.config.DstHost,
+		PeerNatType:    peerNatType,
+		PeerIP:         peerIP,
+		ShareBandwidth: gConf.Network.ShareBandwidth,
+		RelayNode:      relayNode,
+		Version:        OpenP2PVersion,
+	}
+	pn.write(MsgReport, MsgReportConnect, &req)
+	if err != nil {
+		return err
+	}
+	// if rtid != 0 || t.conn.Protocol() == "tcp" {
+	// sync appkey
+	if t == nil {
+		return err
+	}
+	syncKeyReq := APPKeySync{
+		AppID:  app.id,
+		AppKey: app.key,
+	}
+	gLog.Printf(LvDEBUG, "sync appkey direct to %s", app.config.LogPeerNode())
+	pn.push(app.config.PeerNode, MsgPushAPPKey, &syncKeyReq)
+	app.setDirectTunnel(t)
+
+	// if memapp notify peer addmemapp
+	if app.config.SrcPort == 0 {
+		req := ServerSideSaveMemApp{From: gConf.Network.Node, Node: gConf.Network.Node, TunnelID: t.id, RelayTunnelID: 0, AppID: app.id}
+		pn.push(app.config.PeerNode, MsgPushServerSideSaveMemApp, &req)
+		gLog.Printf(LvDEBUG, "push %s ServerSideSaveMemApp: %s", app.config.LogPeerNode(), prettyJson(req))
+	}
+	gLog.Printf(LvDEBUG, "%s use tunnel %d", app.config.AppName, t.id)
+	return nil
+}
+
+func (app *p2pApp) checkRelayTunnel() error {
+	// if app.config.ForceRelay == 1 && (gConf.sdwan.CentralNode == app.config.PeerNode && compareVersion(app.config.peerVersion, SupportDualTunnelVersion) < 0) {
+	if app.config.SrcPort == 0 && (gConf.sdwan.CentralNode == app.config.PeerNode || gConf.sdwan.CentralNode == gConf.Network.Node) { // memapp central node not build relay tunnel
+		return nil
+	}
+	app.hbMtx.Lock()
+	if app.RelayTunnel() != nil && time.Now().Before(app.hbTimeRelay.Add(TunnelHeartbeatTime*2)) { // must check app.hbtime instead of relayTunnel
+		app.hbMtx.Unlock()
+		return nil
+	}
+	app.hbMtx.Unlock()
+	if app.nextRetryRelayTime.After(time.Now()) || app.config.Enabled == 0 || app.retryRelayNum >= retryLimit {
+		return nil
+	}
+	if time.Now().Add(-time.Minute * 15).After(app.retryRelayTime) { // run normally 15min, reset retrynum
+		app.retryRelayNum = 1
+	}
+	if app.retryRelayNum > 0 { // first time not show reconnect log
+		gLog.Printf(LvINFO, "detect app %s appid:%d relay disconnect, reconnecting the %d times...", app.config.LogPeerNode(), app.id, app.retryRelayNum)
+	}
+	app.setRelayTunnel(nil) // reset relayTunnel
+	app.retryRelayNum++
+	app.retryRelayTime = time.Now()
+	app.nextRetryRelayTime = time.Now().Add(retryInterval)
+	app.connectTime = time.Now()
+	err := app.buildRelayTunnel()
+	if err != nil {
+		app.errMsg = err.Error()
+		if err == ErrPeerOffline && app.retryRelayNum > 2 { // stop retry, waiting for online
+			app.retryRelayNum = retryLimit
+			gLog.Printf(LvINFO, " %s offline, it will auto reconnect when peer node online", app.config.LogPeerNode())
+		}
+	}
+	if app.Tunnel() != nil {
+		app.once.Do(func() {
+			go app.listen()
+			// memapp also need
+			go app.relayHeartbeatLoop()
+		})
+	}
+	return nil
+}
+
+func (app *p2pApp) buildRelayTunnel() error {
+	var rtid uint64
+	relayNode := ""
+	relayMode := ""
+	peerNatType := NATUnknown
+	peerIP := ""
+	errMsg := ""
+	var t *P2PTunnel
+	var err error
+	pn := GNetwork
+	config := app.config
+	initErr := pn.requestPeerInfo(&config)
+	if initErr != nil {
+		gLog.Printf(LvERROR, "%s init error:%s", config.LogPeerNode(), initErr)
+		return initErr
+	}
+
+	t, rtid, relayMode, err = pn.addRelayTunnel(config)
+	if t != nil {
+		relayNode = t.config.PeerNode
+	}
+
+	if err != nil {
+		errMsg = err.Error()
+	}
+	req := ReportConnect{
+		Error:          errMsg,
+		Protocol:       config.Protocol,
+		SrcPort:        config.SrcPort,
+		NatType:        gConf.Network.natType,
+		PeerNode:       config.PeerNode,
+		DstPort:        config.DstPort,
+		DstHost:        config.DstHost,
+		PeerNatType:    peerNatType,
+		PeerIP:         peerIP,
+		ShareBandwidth: gConf.Network.ShareBandwidth,
+		RelayNode:      relayNode,
+		Version:        OpenP2PVersion,
+	}
+	pn.write(MsgReport, MsgReportConnect, &req)
+	if err != nil {
+		return err
+	}
+	// if rtid != 0 || t.conn.Protocol() == "tcp" {
+	// sync appkey
+	syncKeyReq := APPKeySync{
+		AppID:  app.id,
+		AppKey: app.key,
+	}
+	gLog.Printf(LvDEBUG, "sync appkey relay to %s", config.LogPeerNode())
+	pn.push(config.PeerNode, MsgPushAPPKey, &syncKeyReq)
+	app.setRelayTunnelID(rtid)
+	app.setRelayTunnel(t)
+	app.relayNode = relayNode
+	app.relayMode = relayMode
+	app.hbTimeRelay = time.Now()
+
+	// if memapp notify peer addmemapp
+	if config.SrcPort == 0 {
+		req := ServerSideSaveMemApp{From: gConf.Network.Node, Node: relayNode, TunnelID: rtid, RelayTunnelID: t.id, AppID: app.id, RelayMode: relayMode}
+		pn.push(config.PeerNode, MsgPushServerSideSaveMemApp, &req)
+		gLog.Printf(LvDEBUG, "push %s relay ServerSideSaveMemApp: %s", config.LogPeerNode(), prettyJson(req))
+	}
+	gLog.Printf(LvDEBUG, "%s use tunnel %d", app.config.AppName, t.id)
+	return nil
+}
+
+func (app *p2pApp) buildOfficialTunnel() error {
+	return nil
+}
+
+// cache relayHead, refresh when rtid change
+func (app *p2pApp) RelayHead() *bytes.Buffer {
+	if app.relayHead == nil {
+		app.relayHead = new(bytes.Buffer)
+		binary.Write(app.relayHead, binary.LittleEndian, app.rtid)
+	}
+	return app.relayHead
+}
+
+func (app *p2pApp) setRelayTunnelID(rtid uint64) {
+	app.rtid = rtid
+	app.relayHead = new(bytes.Buffer)
+	binary.Write(app.relayHead, binary.LittleEndian, app.rtid)
 }
 
 func (app *p2pApp) isActive() bool {
-	if app.tunnel == nil {
+	if app.Tunnel() == nil {
+		// gLog.Printf(LvDEBUG, "isActive app.tunnel==nil")
 		return false
 	}
-	if app.rtid == 0 { // direct mode app heartbeat equals to tunnel heartbeat
+	if app.isDirect() { // direct mode app heartbeat equals to tunnel heartbeat
-		return app.tunnel.isActive()
+		return app.Tunnel().isActive()
 	}
 	// relay mode calc app heartbeat
 	app.hbMtx.Lock()
 	defer app.hbMtx.Unlock()
-	return time.Now().Before(app.hbTime.Add(TunnelIdleTimeout))
+	res := time.Now().Before(app.hbTimeRelay.Add(TunnelHeartbeatTime * 2))
+	// if !res {
+	// 	gLog.Printf(LvDEBUG, "%d app isActive false. peer=%s", app.id, app.config.PeerNode)
+	// }
+	return res
 }
 
 func (app *p2pApp) updateHeartbeat() {
 	app.hbMtx.Lock()
 	defer app.hbMtx.Unlock()
-	app.hbTime = time.Now()
+	app.hbTimeRelay = time.Now()
 }
 
 func (app *p2pApp) listenTCP() error {
 	gLog.Printf(LvDEBUG, "tcp accept on port %d start", app.config.SrcPort)
 	defer gLog.Printf(LvDEBUG, "tcp accept on port %d end", app.config.SrcPort)
 	var err error
-	app.listener, err = net.Listen("tcp4", fmt.Sprintf("0.0.0.0:%d", app.config.SrcPort))
+	listenAddr := ""
+	if IsLocalhost(app.config.Whitelist) { // not expose port
+		listenAddr = "127.0.0.1"
+	}
+	app.listener, err = net.Listen("tcp", fmt.Sprintf("%s:%d", listenAddr, app.config.SrcPort))
 	if err != nil {
 		gLog.Printf(LvERROR, "listen error:%s", err)
 		return err
 	}
+	defer app.listener.Close()
 	for app.running {
 		conn, err := app.listener.Accept()
 		if err != nil {
@@ -64,14 +401,32 @@ func (app *p2pApp) listenTCP() error {
 			}
 			break
 		}
+		if app.Tunnel() == nil {
+			gLog.Printf(LvDEBUG, "srcPort=%d, app.Tunnel()==nil, not ready", app.config.SrcPort)
+			time.Sleep(time.Second)
+			continue
+		}
+		// check white list
+		if app.config.Whitelist != "" {
+			remoteIP := conn.RemoteAddr().(*net.TCPAddr).IP.String()
+			if !app.iptree.Contains(remoteIP) && !IsLocalhost(remoteIP) {
+				conn.Close()
+				gLog.Printf(LvERROR, "%s not in whitelist, access denied", remoteIP)
+				continue
+			}
+		}
 		oConn := overlayConn{
-			tunnel:   app.tunnel,
+			tunnel:   app.Tunnel(),
+			app:      app,
 			connTCP:  conn,
 			id:       rand.Uint64(),
 			isClient: true,
-			rtid:     app.rtid,
 			appID:    app.id,
 			appKey:   app.key,
+			running:  true,
+		}
+		if !app.isDirect() {
+			oConn.rtid = app.rtid
 		}
 		// pre-calc key bytes for encrypt
 		if oConn.appKey != 0 {
@@ -80,26 +435,22 @@ func (app *p2pApp) listenTCP() error {
 			binary.LittleEndian.PutUint64(encryptKey[8:], oConn.appKey)
 			oConn.appKeyBytes = encryptKey
 		}
-		app.tunnel.overlayConns.Store(oConn.id, &oConn)
+		app.Tunnel().overlayConns.Store(oConn.id, &oConn)
-		gLog.Printf(LvDEBUG, "Accept TCP overlayID:%d", oConn.id)
+		gLog.Printf(LvDEBUG, "Accept TCP overlayID:%d, %s", oConn.id, oConn.connTCP.RemoteAddr())
 		// tell peer connect
 		req := OverlayConnectReq{ID: oConn.id,
-			Token:    app.tunnel.pn.config.Token,
+			Token:    gConf.Network.Token,
 			DstIP:    app.config.DstHost,
 			DstPort:  app.config.DstPort,
 			Protocol: app.config.Protocol,
 			AppID:    app.id,
 		}
-		if app.rtid == 0 {
+		if !app.isDirect() {
-			app.tunnel.conn.WriteMessage(MsgP2P, MsgOverlayConnectReq, &req)
+			req.RelayTunnelID = app.Tunnel().id
-		} else {
-			req.RelayTunnelID = app.tunnel.id
-			relayHead := new(bytes.Buffer)
-			binary.Write(relayHead, binary.LittleEndian, app.rtid)
-			msg, _ := newMessage(MsgP2P, MsgOverlayConnectReq, &req)
-			msgWithHead := append(relayHead.Bytes(), msg...)
-			app.tunnel.conn.WriteBytes(MsgP2P, MsgRelayData, msgWithHead)
 		}
+		app.Tunnel().WriteMessage(app.RelayTunnelID(), MsgP2P, MsgOverlayConnectReq, &req)
+		// TODO: wait OverlayConnectRsp instead of sleep
+		time.Sleep(time.Second) // waiting remote node connection ok
 		go oConn.run()
 	}
 	return nil
@@ -114,10 +465,11 @@ func (app *p2pApp) listenUDP() error {
 		gLog.Printf(LvERROR, "listen error:%s", err)
 		return err
 	}
-	buffer := make([]byte, 64*1024)
+	defer app.listenerUDP.Close()
+	buffer := make([]byte, 64*1024+PaddingSize)
 	udpID := make([]byte, 8)
 	for {
-		app.listenerUDP.SetReadDeadline(time.Now().Add(time.Second * 10))
+		app.listenerUDP.SetReadDeadline(time.Now().Add(UDPReadTimeout))
 		len, remoteAddr, err := app.listenerUDP.ReadFrom(buffer)
 		if err != nil {
 			if ne, ok := err.(net.Error); ok && ne.Timeout() {
@@ -127,8 +479,13 @@ func (app *p2pApp) listenUDP() error {
 				break
 			}
 		} else {
-			b := bytes.Buffer{}
+			if app.Tunnel() == nil {
-			b.Write(buffer[:len])
+				gLog.Printf(LvDEBUG, "srcPort=%d, app.Tunnel()==nil, not ready", app.config.SrcPort)
+				time.Sleep(time.Second)
+				continue
+			}
+			dupData := bytes.Buffer{} // should uses memory pool
+			dupData.Write(buffer[:len+PaddingSize])
 			// load from app.tunnel.overlayConns by remoteAddr error, new udp connection
 			remoteIP := strings.Split(remoteAddr.String(), ":")[0]
 			port, _ := strconv.Atoi(strings.Split(remoteAddr.String(), ":")[1])
@@ -139,19 +496,22 @@ func (app *p2pApp) listenUDP() error {
 			udpID[3] = a[3]
 			udpID[4] = byte(port)
 			udpID[5] = byte(port >> 8)
-			id := binary.LittleEndian.Uint64(udpID)
+			id := binary.LittleEndian.Uint64(udpID) // convert remoteIP:port to uint64
-			s, ok := app.tunnel.overlayConns.Load(id)
+			s, ok := app.Tunnel().overlayConns.Load(id)
 			if !ok {
 				oConn := overlayConn{
-					tunnel:       app.tunnel,
+					tunnel:     app.Tunnel(),
-					connUDP:      app.listenerUDP,
+					connUDP:    app.listenerUDP,
-					remoteAddr:   remoteAddr,
+					remoteAddr: remoteAddr,
-					udpRelayData: make(chan []byte, 1000),
+					udpData:    make(chan []byte, 1000),
-					id:           id,
+					id:         id,
-					isClient:     true,
+					isClient:   true,
-					rtid:         app.rtid,
+					appID:      app.id,
-					appID:        app.id,
+					appKey:     app.key,
-					appKey:       app.key,
+					running:    true,
+				}
+				if !app.isDirect() {
+					oConn.rtid = app.rtid
 				}
 				// calc key bytes for encrypt
 				if oConn.appKey != 0 {
@@ -160,28 +520,24 @@ func (app *p2pApp) listenUDP() error {
 					binary.LittleEndian.PutUint64(encryptKey[8:], oConn.appKey)
 					oConn.appKeyBytes = encryptKey
 				}
-				app.tunnel.overlayConns.Store(oConn.id, &oConn)
+				app.Tunnel().overlayConns.Store(oConn.id, &oConn)
 				gLog.Printf(LvDEBUG, "Accept UDP overlayID:%d", oConn.id)
 				// tell peer connect
 				req := OverlayConnectReq{ID: oConn.id,
-					Token:    app.tunnel.pn.config.Token,
+					Token:    gConf.Network.Token,
 					DstIP:    app.config.DstHost,
 					DstPort:  app.config.DstPort,
 					Protocol: app.config.Protocol,
 					AppID:    app.id,
 				}
-				if app.rtid == 0 {
+				if !app.isDirect() {
-					app.tunnel.conn.WriteMessage(MsgP2P, MsgOverlayConnectReq, &req)
+					req.RelayTunnelID = app.Tunnel().id
-				} else {
-					req.RelayTunnelID = app.tunnel.id
-					relayHead := new(bytes.Buffer)
-					binary.Write(relayHead, binary.LittleEndian, app.rtid)
-					msg, _ := newMessage(MsgP2P, MsgOverlayConnectReq, &req)
-					msgWithHead := append(relayHead.Bytes(), msg...)
-					app.tunnel.conn.WriteBytes(MsgP2P, MsgRelayData, msgWithHead)
 				}
+				app.Tunnel().WriteMessage(app.RelayTunnelID(), MsgP2P, MsgOverlayConnectReq, &req)
+				// TODO: wait OverlayConnectRsp instead of sleep
+				time.Sleep(time.Second) // waiting remote node connection ok
 				go oConn.run()
-				oConn.udpRelayData <- b.Bytes()
+				oConn.udpData <- dupData.Bytes()
 			}
 
 			// load from app.tunnel.overlayConns by remoteAddr ok, write relay data
@@ -189,22 +545,21 @@ func (app *p2pApp) listenUDP() error {
 			if !ok {
 				continue
 			}
-			overlayConn.udpRelayData <- b.Bytes()
+			overlayConn.udpData <- dupData.Bytes()
 		}
 	}
 	return nil
 }
 
 func (app *p2pApp) listen() error {
+	if app.config.SrcPort == 0 {
+		return nil
+	}
 	gLog.Printf(LvINFO, "LISTEN ON PORT %s:%d START", app.config.Protocol, app.config.SrcPort)
 	defer gLog.Printf(LvINFO, "LISTEN ON PORT %s:%d END", app.config.Protocol, app.config.SrcPort)
 	app.wg.Add(1)
 	defer app.wg.Done()
-	app.running = true
+	for app.running {
-	if app.rtid != 0 {
-		go app.relayHeartbeatLoop()
-	}
-	for app.tunnel.isRuning() {
 		if app.config.Protocol == "udp" {
 			app.listenUDP()
 		} else {
@@ -226,8 +581,11 @@ func (app *p2pApp) close() {
 	if app.listenerUDP != nil {
 		app.listenerUDP.Close()
 	}
-	if app.tunnel != nil {
+	if app.DirectTunnel() != nil {
-		app.tunnel.closeOverlayConns(app.id)
+		app.DirectTunnel().closeOverlayConns(app.id)
+	}
+	if app.RelayTunnel() != nil {
+		app.RelayTunnel().closeOverlayConns(app.id)
 	}
 	app.wg.Wait()
 }
@@ -236,16 +594,23 @@ func (app *p2pApp) close() {
 func (app *p2pApp) relayHeartbeatLoop() {
 	app.wg.Add(1)
 	defer app.wg.Done()
-	gLog.Printf(LvDEBUG, "relayHeartbeat to %d start", app.rtid)
+	gLog.Printf(LvDEBUG, "%s appid:%d relayHeartbeat to rtid:%d start", app.config.LogPeerNode(), app.id, app.rtid)
-	defer gLog.Printf(LvDEBUG, "relayHeartbeat to %d end", app.rtid)
+	defer gLog.Printf(LvDEBUG, "%s appid:%d relayHeartbeat to rtid%d end", app.config.LogPeerNode(), app.id, app.rtid)
-	relayHead := new(bytes.Buffer)
+
-	binary.Write(relayHead, binary.LittleEndian, app.rtid)
+	for app.running {
-	req := RelayHeartbeat{RelayTunnelID: app.tunnel.id,
+		if app.RelayTunnel() == nil || !app.RelayTunnel().isRuning() {
-		AppID: app.id}
+			time.Sleep(TunnelHeartbeatTime)
-	msg, _ := newMessage(MsgP2P, MsgRelayHeartbeat, &req)
+			continue
-	msgWithHead := append(relayHead.Bytes(), msg...)
+		}
-	for app.tunnel.isRuning() && app.running {
+		req := RelayHeartbeat{From: gConf.Network.Node, RelayTunnelID: app.RelayTunnel().id,
-		app.tunnel.conn.WriteBytes(MsgP2P, MsgRelayData, msgWithHead)
+			AppID: app.id}
+		err := app.RelayTunnel().WriteMessage(app.rtid, MsgP2P, MsgRelayHeartbeat, &req)
+		if err != nil {
+			gLog.Printf(LvERROR, "%s appid:%d rtid:%d write relay tunnel heartbeat error %s", app.config.LogPeerNode(), app.id, app.rtid, err)
+			return
+		}
+		// TODO: debug relay heartbeat
+		gLog.Printf(LvDEBUG, "%s appid:%d rtid:%d write relay tunnel heartbeat ok", app.config.LogPeerNode(), app.id, app.rtid)
 		time.Sleep(TunnelHeartbeatTime)
 	}
 }

core/p2pnetwork.go

@@ -1,715 +1,976 @@
-package openp2p
+package openp2p
-
+
-import (
+import (
-	"bytes"
+	"bytes"
-	"crypto/tls"
+	"crypto/tls"
-	"encoding/binary"
+	"crypto/x509"
-	"encoding/json"
+	"encoding/binary"
-	"errors"
+	"encoding/json"
-	"fmt"
+	"errors"
-	"math/rand"
+	"fmt"
-	"net/http"
+	"math/rand"
-	"net/url"
+	"net/http"
-	"strings"
+	"net/url"
-	"sync"
+	"reflect"
-	"time"
+	"strings"
-
+	"sync"
-	"github.com/gorilla/websocket"
+	"time"
-)
+
-
+	"github.com/gorilla/websocket"
-var (
+)
-	instance *P2PNetwork
+
-	once     sync.Once
+var (
-)
+	v4l            *v4Listener
-
+	instance       *P2PNetwork
-type P2PNetwork struct {
+	onceP2PNetwork sync.Once
-	conn        *websocket.Conn
+	onceV4Listener sync.Once
-	online      bool
+)
-	running     bool
+
-	restartCh   chan bool
+const (
-	wgReconnect sync.WaitGroup
+	retryLimit                  = 20
-	writeMtx    sync.Mutex
+	retryInterval               = 10 * time.Second
-	serverTs    int64
+	DefaultLoginMaxDelaySeconds = 60
-	localTs     int64
+)
-	hbTime      time.Time
+
-	// msgMap    sync.Map
+// golang not support float64 const
-	msgMap     map[uint64]chan []byte //key: nodeID
+var (
-	msgMapMtx  sync.Mutex
+	ma10 float64 = 1.0 / 10
-	config     NetworkConfig
+	ma5  float64 = 1.0 / 5
-	allTunnels sync.Map
+)
-	apps       sync.Map //key: protocol+srcport; value: p2pApp
+
-	limiter    *BandwidthLimiter
+type NodeData struct {
-}
+	NodeID uint64
-
+	Data   []byte
-func P2PNetworkInstance(config *NetworkConfig) *P2PNetwork {
+}
-	if instance == nil {
+
-		once.Do(func() {
+type P2PNetwork struct {
-			instance = &P2PNetwork{
+	conn          *websocket.Conn
-				restartCh: make(chan bool, 2),
+	online        bool
-				online:    false,
+	running       bool
-				running:   true,
+	restartCh     chan bool
-				msgMap:    make(map[uint64]chan []byte),
+	wgReconnect   sync.WaitGroup
-				limiter:   newBandwidthLimiter(config.ShareBandwidth),
+	writeMtx      sync.Mutex
-			}
+	reqGatewayMtx sync.Mutex
-			instance.msgMap[0] = make(chan []byte) // for gateway
+	hbTime        time.Time
-			if config != nil {
+	// for sync server time
-				instance.config = *config
+	t1     int64 // nanoSeconds
-			}
+	preRtt int64 // nanoSeconds
-			instance.init()
+	dt     int64 // client faster then server dt nanoSeconds
-			go instance.run()
+	ddtma  int64
-		})
+	ddt    int64    // differential of dt
-	}
+	msgMap sync.Map //key: nodeID
-	return instance
+	// msgMap     map[uint64]chan pushMsg //key: nodeID
-}
+	allTunnels           sync.Map // key: tid
-
+	apps                 sync.Map //key: config.ID(); value: *p2pApp
-func (pn *P2PNetwork) run() {
+	limiter              *SpeedLimiter
-	go pn.autorunApp()
+	nodeData             chan *NodeData
-	heartbeatTimer := time.NewTicker(NetworkHeartbeatTime)
+	sdwan                *p2pSDWAN
-	for pn.running {
+	tunnelCloseCh        chan *P2PTunnel
-		select {
+	loginMaxDelaySeconds int
-		case <-heartbeatTimer.C:
+}
-			pn.write(MsgHeartbeat, 0, "")
+
-
+type msgCtx struct {
-		case <-pn.restartCh:
+	data []byte
-			pn.online = false
+	ts   time.Time
-			pn.wgReconnect.Wait() // wait read/write goroutine end
+}
-			err := pn.init()
+
-			if err != nil {
+func P2PNetworkInstance() *P2PNetwork {
-				gLog.Println(LvERROR, "P2PNetwork init error:", err)
+	if instance == nil {
-			}
+		onceP2PNetwork.Do(func() {
-		}
+			instance = &P2PNetwork{
-	}
+				restartCh:            make(chan bool, 1),
-}
+				tunnelCloseCh:        make(chan *P2PTunnel, 100),
-
+				nodeData:             make(chan *NodeData, 10000),
-func (pn *P2PNetwork) Connect(timeout int) bool {
+				online:               false,
-	// waiting for login response
+				running:              true,
-	for i := 0; i < (timeout / 1000); i++ {
+				limiter:              newSpeedLimiter(gConf.Network.ShareBandwidth*1024*1024/8, 1),
-		if pn.hbTime.After(time.Now().Add(-NetworkHeartbeatTime)) {
+				dt:                   0,
-			return true
+				ddt:                  0,
-		}
+				loginMaxDelaySeconds: DefaultLoginMaxDelaySeconds,
-		time.Sleep(time.Second)
+			}
-	}
+			instance.msgMap.Store(uint64(0), make(chan msgCtx, 50)) // for gateway
-	return false
+			instance.StartSDWAN()
-}
+			instance.init()
-
+			go instance.run()
-func (pn *P2PNetwork) runAll() {
+			go func() {
-	gConf.mtx.Lock() // lock for copy gConf.Apps and the modification of config(it's pointer)
+				for {
-	defer gConf.mtx.Unlock()
+					instance.refreshIPv6()
-	allApps := gConf.Apps // read a copy, other thread will modify the gConf.Apps
+					time.Sleep(time.Hour)
-
+				}
-	for _, config := range allApps {
+			}()
-		if config.nextRetryTime.After(time.Now()) {
+			cleanTempFiles()
-			continue
+		})
-		}
+	}
-		if config.Enabled == 0 {
+	return instance
-			continue
+}
-		}
+
-		if config.AppName == "" {
+func (pn *P2PNetwork) run() {
-			config.AppName = fmt.Sprintf("%s%d", config.Protocol, config.SrcPort)
+	heartbeatTimer := time.NewTicker(NetworkHeartbeatTime)
-		}
+	pn.t1 = time.Now().UnixNano()
-		appExist := false
+	pn.write(MsgHeartbeat, 0, "")
-		i, ok := pn.apps.Load(fmt.Sprintf("%s%d", config.Protocol, config.SrcPort))
+	for {
-		if ok {
+		select {
-			app := i.(*p2pApp)
+		case <-heartbeatTimer.C:
-			appExist = true
+			pn.t1 = time.Now().UnixNano()
-			if app.isActive() {
+			pn.write(MsgHeartbeat, 0, "")
-				continue
+		case <-pn.restartCh:
-			}
+			gLog.Printf(LvDEBUG, "got restart channel")
-		}
+			pn.online = false
-		if appExist {
+			pn.wgReconnect.Wait() // wait read/autorunapp goroutine end
-			pn.DeleteApp(*config)
+			delay := ClientAPITimeout + time.Duration(rand.Int()%pn.loginMaxDelaySeconds)*time.Second
-		}
+			time.Sleep(delay)
-		if config.retryNum > 0 {
+			err := pn.init()
-			gLog.Printf(LvINFO, "detect app %s disconnect, reconnecting the %d times...", config.AppName, config.retryNum)
+			if err != nil {
-			if time.Now().Add(-time.Minute * 15).After(config.retryTime) { // normal lasts 15min
+				gLog.Println(LvERROR, "P2PNetwork init error:", err)
-				config.retryNum = 0
+			}
-			}
+			gConf.retryAllApp()
-		}
+		case t := <-pn.tunnelCloseCh:
-		config.retryNum++
+			gLog.Printf(LvDEBUG, "got tunnelCloseCh %s", t.config.LogPeerNode())
-		config.retryTime = time.Now()
+			pn.apps.Range(func(id, i interface{}) bool {
-		if config.retryNum > 20 {
+				app := i.(*p2pApp)
-			config.Enabled = 0
+				if app.DirectTunnel() == t {
-			gLog.Printf(LvWARN, "app %s has stopped retry, manually enable it on Web console", config.AppName)
+					app.setDirectTunnel(nil)
-			continue
+				}
-		}
+				if app.RelayTunnel() == t {
-		config.nextRetryTime = time.Now().Add(time.Second * 10)
+					app.setRelayTunnel(nil)
-		config.connectTime = time.Now()
+				}
-		config.peerToken = pn.config.Token
+				return true
-		gConf.mtx.Unlock() // AddApp will take a period of time
+			})
-		err := pn.AddApp(*config)
+		}
-		gConf.mtx.Lock()
+	}
-		if err != nil {
+}
-			config.errMsg = err.Error()
+
-		}
+func (pn *P2PNetwork) NotifyTunnelClose(t *P2PTunnel) bool {
-	}
+	select {
-}
+	case pn.tunnelCloseCh <- t:
-func (pn *P2PNetwork) autorunApp() {
+		return true
-	gLog.Println(LvINFO, "autorunApp start")
+	default:
-	for pn.running {
+	}
-		time.Sleep(time.Second)
+	return false
-		if !pn.online {
+}
-			continue
+
-		}
+func (pn *P2PNetwork) Connect(timeout int) bool {
-		pn.runAll()
+	// waiting for heartbeat
-	}
+	for i := 0; i < (timeout / 1000); i++ {
-	gLog.Println(LvINFO, "autorunApp end")
+		if pn.hbTime.After(time.Now().Add(-NetworkHeartbeatTime)) {
-}
+			return true
-
+		}
-func (pn *P2PNetwork) addRelayTunnel(config AppConfig) (*P2PTunnel, uint64, string, error) {
+		time.Sleep(time.Second)
-	gLog.Printf(LvINFO, "addRelayTunnel to %s start", config.PeerNode)
+	}
-	defer gLog.Printf(LvINFO, "addRelayTunnel to %s end", config.PeerNode)
+	return false
-	// request a relay node or specify manually(TODO)
+}
-	pn.write(MsgRelay, MsgRelayNodeReq, &RelayNodeReq{config.PeerNode})
+
-	head, body := pn.read("", MsgRelay, MsgRelayNodeRsp, time.Second*10)
+func (pn *P2PNetwork) runAll() {
-	if head == nil {
+	gConf.mtx.Lock() // lock for copy gConf.Apps and the modification of config(it's pointer)
-		return nil, 0, "", errors.New("read MsgRelayNodeRsp error")
+	defer gConf.mtx.Unlock()
-	}
+	allApps := gConf.Apps // read a copy, other thread will modify the gConf.Apps
-	rsp := RelayNodeRsp{}
+	for _, config := range allApps {
-	err := json.Unmarshal(body, &rsp)
+		if config.AppName == "" {
-	if err != nil {
+			config.AppName = fmt.Sprintf("%d", config.ID())
-		gLog.Printf(LvERROR, "wrong RelayNodeRsp:%s", err)
+		}
-		return nil, 0, "", errors.New("unmarshal MsgRelayNodeRsp error")
+		if config.Enabled == 0 {
-	}
+			continue
-	if rsp.RelayName == "" || rsp.RelayToken == 0 {
+		}
-		gLog.Printf(LvERROR, "MsgRelayNodeReq error")
+		if _, ok := pn.apps.Load(config.ID()); ok {
-		return nil, 0, "", errors.New("MsgRelayNodeReq error")
+			continue
-	}
+		}
-	gLog.Printf(LvINFO, "got relay node:%s", rsp.RelayName)
+
-	relayConfig := config
+		config.peerToken = gConf.Network.Token
-	relayConfig.PeerNode = rsp.RelayName
+		gConf.mtx.Unlock() // AddApp will take a period of time, let outside modify gConf
-	relayConfig.peerToken = rsp.RelayToken
+		pn.AddApp(*config)
-	///
+		gConf.mtx.Lock()
-	t, err := pn.addDirectTunnel(relayConfig, 0)
+
-	if err != nil {
+	}
-		gLog.Println(LvERROR, "direct connect error:", err)
+}
-		return nil, 0, "", err
+
-	}
+func (pn *P2PNetwork) autorunApp() {
-	// notify peer addRelayTunnel
+	gLog.Println(LvINFO, "autorunApp start")
-	req := AddRelayTunnelReq{
+	pn.wgReconnect.Add(1)
-		From:       pn.config.Node,
+	defer pn.wgReconnect.Done()
-		RelayName:  rsp.RelayName,
+	for pn.running && pn.online {
-		RelayToken: rsp.RelayToken,
+		time.Sleep(time.Second)
-	}
+		pn.runAll()
-	gLog.Printf(LvINFO, "push relay %s---------%s", config.PeerNode, rsp.RelayName)
+	}
-	pn.push(config.PeerNode, MsgPushAddRelayTunnelReq, &req)
+	gLog.Println(LvINFO, "autorunApp end")
-
+}
-	// wait relay ready
+
-	head, body = pn.read(config.PeerNode, MsgPush, MsgPushAddRelayTunnelRsp, PeerAddRelayTimeount) // TODO: const value
+func (pn *P2PNetwork) addRelayTunnel(config AppConfig) (*P2PTunnel, uint64, string, error) {
-	if head == nil {
+	gLog.Printf(LvINFO, "addRelayTunnel to %s start", config.LogPeerNode())
-		gLog.Printf(LvERROR, "read MsgPushAddRelayTunnelRsp error")
+	defer gLog.Printf(LvINFO, "addRelayTunnel to %s end", config.LogPeerNode())
-		return nil, 0, "", errors.New("read MsgPushAddRelayTunnelRsp error")
+	relayConfig := AppConfig{
-	}
+		PeerNode:  config.RelayNode,
-	rspID := TunnelMsg{}
+		peerToken: config.peerToken,
-	err = json.Unmarshal(body, &rspID)
+		relayMode: "private"}
-	if err != nil {
+	if relayConfig.PeerNode == "" {
-		gLog.Printf(LvERROR, "wrong RelayNodeRsp:%s", err)
+		// find existing relay tunnel
-		return nil, 0, "", errors.New("unmarshal MsgRelayNodeRsp error")
+		pn.apps.Range(func(id, i interface{}) bool {
-	}
+			app := i.(*p2pApp)
-	return t, rspID.ID, rsp.Mode, err
+			if app.config.PeerNode != config.PeerNode {
-}
+				return true
-
+			}
-// use *AppConfig to save status
+			if app.RelayTunnel() == nil {
-func (pn *P2PNetwork) AddApp(config AppConfig) error {
+				return true
-	gLog.Printf(LvINFO, "addApp %s to %s:%s:%d start", config.AppName, config.PeerNode, config.DstHost, config.DstPort)
+			}
-	defer gLog.Printf(LvINFO, "addApp %s to %s:%s:%d end", config.AppName, config.PeerNode, config.DstHost, config.DstPort)
+			relayConfig.PeerNode = app.RelayTunnel().config.PeerNode
-	if !pn.online {
+			gLog.Printf(LvDEBUG, "found existing relay tunnel %s", relayConfig.LogPeerNode())
-		return errors.New("P2PNetwork offline")
+			return false
-	}
+		})
-	// check if app already exist?
+		if relayConfig.PeerNode == "" { // request relay node
-	appExist := false
+			pn.reqGatewayMtx.Lock()
-	_, ok := pn.apps.Load(fmt.Sprintf("%s%d", config.Protocol, config.SrcPort))
+			pn.write(MsgRelay, MsgRelayNodeReq, &RelayNodeReq{config.PeerNode})
-	if ok {
+			head, body := pn.read("", MsgRelay, MsgRelayNodeRsp, ClientAPITimeout)
-		appExist = true
+			pn.reqGatewayMtx.Unlock()
-	}
+			if head == nil {
-	if appExist {
+				return nil, 0, "", errors.New("read MsgRelayNodeRsp error")
-		return errors.New("P2PApp already exist")
+			}
-	}
+			rsp := RelayNodeRsp{}
-	appID := rand.Uint64()
+			if err := json.Unmarshal(body, &rsp); err != nil {
-	appKey := uint64(0)
+				return nil, 0, "", errors.New("unmarshal MsgRelayNodeRsp error")
-	var rtid uint64
+			}
-	relayNode := ""
+			if rsp.RelayName == "" || rsp.RelayToken == 0 {
-	relayMode := ""
+				gLog.Printf(LvERROR, "MsgRelayNodeReq error")
-	peerNatType := NATUnknown
+				return nil, 0, "", errors.New("MsgRelayNodeReq error")
-	peerIP := ""
+			}
-	errMsg := ""
+			gLog.Printf(LvDEBUG, "got relay node:%s", relayConfig.LogPeerNode())
-	t, err := pn.addDirectTunnel(config, 0)
+
-	if t != nil {
+			relayConfig.PeerNode = rsp.RelayName
-		peerNatType = t.config.peerNatType
+			relayConfig.peerToken = rsp.RelayToken
-		peerIP = t.config.peerIP
+			relayConfig.relayMode = rsp.Mode
-	}
+		}
-	// TODO: if tcp failed, should try udp punching, nattype should refactor also, when NATNONE and failed we don't know the peerNatType
+
-
+	}
-	if err != nil && err == ErrorHandshake {
+	///
-		gLog.Println(LvERROR, "direct connect failed, try to relay")
+	t, err := pn.addDirectTunnel(relayConfig, 0)
-		t, rtid, relayMode, err = pn.addRelayTunnel(config)
+	if err != nil {
-		if t != nil {
+		gLog.Println(LvERROR, "direct connect error:", err)
-			relayNode = t.config.PeerNode
+		return nil, 0, "", ErrConnectRelayNode // relay offline will stop retry
-		}
+	}
-	}
+	// notify peer addRelayTunnel
-
+	req := AddRelayTunnelReq{
-	if err != nil {
+		From:          gConf.Network.Node,
-		errMsg = err.Error()
+		RelayName:     relayConfig.PeerNode,
-	}
+		RelayToken:    relayConfig.peerToken,
-	req := ReportConnect{
+		RelayMode:     relayConfig.relayMode,
-		Error:          errMsg,
+		RelayTunnelID: t.id,
-		Protocol:       config.Protocol,
+	}
-		SrcPort:        config.SrcPort,
+	gLog.Printf(LvDEBUG, "push %s the relay node(%s)", config.LogPeerNode(), relayConfig.LogPeerNode())
-		NatType:        pn.config.natType,
+	pn.push(config.PeerNode, MsgPushAddRelayTunnelReq, &req)
-		PeerNode:       config.PeerNode,
+
-		DstPort:        config.DstPort,
+	// wait relay ready
-		DstHost:        config.DstHost,
+	head, body := pn.read(config.PeerNode, MsgPush, MsgPushAddRelayTunnelRsp, PeerAddRelayTimeount)
-		PeerNatType:    peerNatType,
+	if head == nil {
-		PeerIP:         peerIP,
+		gLog.Printf(LvERROR, "read MsgPushAddRelayTunnelRsp error")
-		ShareBandwidth: pn.config.ShareBandwidth,
+		return nil, 0, "", errors.New("read MsgPushAddRelayTunnelRsp error")
-		RelayNode:      relayNode,
+	}
-		Version:        OpenP2PVersion,
+	rspID := TunnelMsg{}
-	}
+	if err = json.Unmarshal(body, &rspID); err != nil {
-	pn.write(MsgReport, MsgReportConnect, &req)
+		gLog.Println(LvDEBUG, ErrPeerConnectRelay)
-	if err != nil {
+		return nil, 0, "", ErrPeerConnectRelay
-		return err
+	}
-	}
+	return t, rspID.ID, relayConfig.relayMode, err
-	if rtid != 0 || t.conn.Protocol() == "tcp" {
+}
-		// sync appkey
+
-		appKey = rand.Uint64()
+// use *AppConfig to save status
-		req := APPKeySync{
+func (pn *P2PNetwork) AddApp(config AppConfig) error {
-			AppID:  appID,
+	gLog.Printf(LvINFO, "addApp %s to %s:%s:%d start", config.AppName, config.LogPeerNode(), config.DstHost, config.DstPort)
-			AppKey: appKey,
+	defer gLog.Printf(LvINFO, "addApp %s to %s:%s:%d end", config.AppName, config.LogPeerNode(), config.DstHost, config.DstPort)
-		}
+	if !pn.online {
-		gLog.Printf(LvINFO, "sync appkey to %s", config.PeerNode)
+		return errors.New("P2PNetwork offline")
-		pn.push(config.PeerNode, MsgPushAPPKey, &req)
+	}
-	}
+	if _, ok := pn.msgMap.Load(NodeNameToID(config.PeerNode)); !ok {
-	app := p2pApp{
+		pn.msgMap.Store(NodeNameToID(config.PeerNode), make(chan msgCtx, 50))
-		id:        appID,
+	}
-		key:       appKey,
+	// check if app already exist?
-		tunnel:    t,
+	if _, ok := pn.apps.Load(config.ID()); ok {
-		config:    config,
+		return errors.New("P2PApp already exist")
-		rtid:      rtid,
+	}
-		relayNode: relayNode,
+
-		relayMode: relayMode,
+	app := p2pApp{
-		hbTime:    time.Now()}
+		// tunnel:    t,
-	pn.apps.Store(fmt.Sprintf("%s%d", config.Protocol, config.SrcPort), &app)
+		id:          rand.Uint64(),
-	if err == nil {
+		key:         rand.Uint64(),
-		go app.listen()
+		config:      config,
-	}
+		iptree:      NewIPTree(config.Whitelist),
-	return err
+		running:     true,
-}
+		hbTimeRelay: time.Now(),
-
+	}
-func (pn *P2PNetwork) DeleteApp(config AppConfig) {
+	if _, ok := pn.msgMap.Load(NodeNameToID(config.PeerNode)); !ok {
-	gLog.Printf(LvINFO, "DeleteApp %s%d start", config.Protocol, config.SrcPort)
+		pn.msgMap.Store(NodeNameToID(config.PeerNode), make(chan msgCtx, 50))
-	defer gLog.Printf(LvINFO, "DeleteApp %s%d end", config.Protocol, config.SrcPort)
+	}
-	// close the apps of this config
+	pn.apps.Store(config.ID(), &app)
-	i, ok := pn.apps.Load(fmt.Sprintf("%s%d", config.Protocol, config.SrcPort))
+	gLog.Printf(LvDEBUG, "Store app %d", config.ID())
-	if ok {
+	go app.checkP2PTunnel()
-		app := i.(*p2pApp)
+	return nil
-		gLog.Printf(LvINFO, "app %s exist, delete it", fmt.Sprintf("%s%d", config.Protocol, config.SrcPort))
+}
-		app.close()
+
-		pn.apps.Delete(fmt.Sprintf("%s%d", config.Protocol, config.SrcPort))
+func (pn *P2PNetwork) DeleteApp(config AppConfig) {
-	}
+	gLog.Printf(LvINFO, "DeleteApp %s to %s:%s:%d start", config.AppName, config.LogPeerNode(), config.DstHost, config.DstPort)
-}
+	defer gLog.Printf(LvINFO, "DeleteApp %s to %s:%s:%d end", config.AppName, config.LogPeerNode(), config.DstHost, config.DstPort)
-
+	// close the apps of this config
-func (pn *P2PNetwork) addDirectTunnel(config AppConfig, tid uint64) (*P2PTunnel, error) {
+	i, ok := pn.apps.Load(config.ID())
-	gLog.Printf(LvDEBUG, "addDirectTunnel %s%d to %s:%s:%d start", config.Protocol, config.SrcPort, config.PeerNode, config.DstHost, config.DstPort)
+	if ok {
-	defer gLog.Printf(LvDEBUG, "addDirectTunnel %s%d to %s:%s:%d end", config.Protocol, config.SrcPort, config.PeerNode, config.DstHost, config.DstPort)
+		app := i.(*p2pApp)
-	isClient := false
+		gLog.Printf(LvINFO, "app %s exist, delete it", app.config.AppName)
-	// client side tid=0, assign random uint64
+		app.close()
-	if tid == 0 {
+		pn.apps.Delete(config.ID())
-		tid = rand.Uint64()
+	}
-		isClient = true
+}
-	}
+
-	exist := false
+func (pn *P2PNetwork) findTunnel(peerNode string) (t *P2PTunnel) {
-	// find existing tunnel to peer
+	t = nil
-	var t *P2PTunnel
+	// find existing tunnel to peer
-	pn.allTunnels.Range(func(id, i interface{}) bool {
+	pn.allTunnels.Range(func(id, i interface{}) bool {
-		t = i.(*P2PTunnel)
+		tmpt := i.(*P2PTunnel)
-		if t.config.PeerNode == config.PeerNode {
+		if tmpt.config.PeerNode == peerNode {
-			// server side force close existing tunnel
+			gLog.Println(LvINFO, "tunnel already exist ", peerNode)
-			if !isClient {
+			isActive := tmpt.checkActive()
-				t.close()
+			// inactive, close it
-				return false
+			if !isActive {
-			}
+				gLog.Println(LvINFO, "but it's not active, close it ", peerNode)
-
+				tmpt.close()
-			// client side checking
+			} else {
-			gLog.Println(LvINFO, "tunnel already exist ", config.PeerNode)
+				t = tmpt
-			isActive := t.checkActive()
+			}
-			// inactive, close it
+			return false
-			if !isActive {
+		}
-				gLog.Println(LvINFO, "but it's not active, close it ", config.PeerNode)
+		return true
-				t.close()
+	})
-			} else {
+	return t
-				// active
+}
-				exist = true
+
-			}
+func (pn *P2PNetwork) addDirectTunnel(config AppConfig, tid uint64) (t *P2PTunnel, err error) {
-			return false
+	gLog.Printf(LvDEBUG, "addDirectTunnel %s%d to %s:%s:%d tid:%d start", config.Protocol, config.SrcPort, config.LogPeerNode(), config.DstHost, config.DstPort, tid)
-		}
+	defer gLog.Printf(LvDEBUG, "addDirectTunnel %s%d to %s:%s:%d tid:%d end", config.Protocol, config.SrcPort, config.LogPeerNode(), config.DstHost, config.DstPort, tid)
-		return true
+	isClient := false
-	})
+	// client side tid=0, assign random uint64
-	if exist {
+	if tid == 0 {
-		return t, nil
+		tid = rand.Uint64()
-	}
+		isClient = true
-	// create tunnel if not exist
+	}
-	t = &P2PTunnel{pn: pn,
+	if _, ok := pn.msgMap.Load(NodeNameToID(config.PeerNode)); !ok {
-		config: config,
+		pn.msgMap.Store(NodeNameToID(config.PeerNode), make(chan msgCtx, 50))
-		id:     tid,
+	}
-	}
+
-	pn.msgMapMtx.Lock()
+	// server side
-	pn.msgMap[nodeNameToID(config.PeerNode)] = make(chan []byte, 50)
+	if !isClient {
-	pn.msgMapMtx.Unlock()
+		t, err = pn.newTunnel(config, tid, isClient)
-	// server side
+		return t, err // always return
-	if !isClient {
+	}
-		err := pn.newTunnel(t, tid, isClient)
+	// client side
-		return t, err // always return
+	// peer info
-	}
+	initErr := pn.requestPeerInfo(&config)
-	// client side
+	if initErr != nil {
-	// peer info
+		gLog.Printf(LvERROR, "%s init error:%s", config.LogPeerNode(), initErr)
-	initErr := t.requestPeerInfo()
+
-	if initErr != nil {
+		return nil, initErr
-		gLog.Println(LvERROR, "init error:", initErr)
+	}
-		return nil, initErr
+	gLog.Printf(LvDEBUG, "config.peerNode=%s,config.peerVersion=%s,config.peerIP=%s,config.peerLanIP=%s,gConf.Network.publicIP=%s,config.peerIPv6=%s,config.hasIPv4=%d,config.hasUPNPorNATPMP=%d,gConf.Network.hasIPv4=%d,gConf.Network.hasUPNPorNATPMP=%d,config.peerNatType=%d,gConf.Network.natType=%d,",
-	}
+		config.LogPeerNode(), config.peerVersion, config.peerIP, config.peerLanIP, gConf.Network.publicIP, config.peerIPv6, config.hasIPv4, config.hasUPNPorNATPMP, gConf.Network.hasIPv4, gConf.Network.hasUPNPorNATPMP, config.peerNatType, gConf.Network.natType)
-	var err error
+	// try Intranet
-	// try TCP6
+	if config.peerIP == gConf.Network.publicIP && compareVersion(config.peerVersion, SupportIntranetVersion) >= 0 { // old version client has no peerLanIP
-	if IsIPv6(t.config.peerIPv6) && IsIPv6(t.pn.config.publicIPv6) {
+		gLog.Println(LvINFO, "try Intranet")
-		gLog.Println(LvINFO, "try TCP6")
+		config.linkMode = LinkModeIntranet
-		t.config.linkMode = LinkModeTCP6
+		config.isUnderlayServer = 0
-		t.config.isUnderlayServer = 0
+		if t, err = pn.newTunnel(config, tid, isClient); err == nil {
-		if err = pn.newTunnel(t, tid, isClient); err == nil {
+			return t, nil
-			return t, nil
+		}
-		}
+	}
-	}
+	// try TCP6
-
+	if IsIPv6(config.peerIPv6) && IsIPv6(gConf.IPv6()) {
-	// TODO: try UDP6
+		gLog.Println(LvINFO, "try TCP6")
-
+		config.linkMode = LinkModeTCP6
-	// try TCP4
+		config.isUnderlayServer = 0
-	if t.config.hasIPv4 == 1 || t.pn.config.hasIPv4 == 1 || t.config.hasUPNPorNATPMP == 1 || t.pn.config.hasUPNPorNATPMP == 1 {
+		if t, err = pn.newTunnel(config, tid, isClient); err == nil {
-		gLog.Println(LvINFO, "try TCP4")
+			return t, nil
-		t.config.linkMode = LinkModeTCP4
+		}
-		if t.config.hasIPv4 == 1 || t.config.hasUPNPorNATPMP == 1 {
+	}
-			t.config.isUnderlayServer = 0
+
-		} else {
+	// try UDP6? maybe no
-			t.config.isUnderlayServer = 1
+
-		}
+	// try TCP4
-		if err = pn.newTunnel(t, tid, isClient); err == nil {
+	if config.hasIPv4 == 1 || gConf.Network.hasIPv4 == 1 || config.hasUPNPorNATPMP == 1 || gConf.Network.hasUPNPorNATPMP == 1 {
-			return t, nil
+		gLog.Println(LvINFO, "try TCP4")
-		}
+		config.linkMode = LinkModeTCP4
-	}
+		if gConf.Network.hasIPv4 == 1 || gConf.Network.hasUPNPorNATPMP == 1 {
-	// TODO: try UDP4
+			config.isUnderlayServer = 1
-
+		} else {
-	// try TCPPunch
+			config.isUnderlayServer = 0
-	if t.config.peerNatType == NATCone && t.pn.config.natType == NATCone { // TODO: support c2s
+		}
-		gLog.Println(LvINFO, "try TCP4 Punch")
+		if t, err = pn.newTunnel(config, tid, isClient); err == nil {
-		t.config.linkMode = LinkModeTCPPunch
+			return t, nil
-		t.config.isUnderlayServer = 0
+		}
-		if err = pn.newTunnel(t, tid, isClient); err == nil {
+	}
-			return t, nil
+	// try UDP4? maybe no
-		}
+	var primaryPunchFunc func() (*P2PTunnel, error)
-	}
+	var secondaryPunchFunc func() (*P2PTunnel, error)
-	// try UDPPunch
+	funcUDP := func() (t *P2PTunnel, err error) {
-	if t.config.peerNatType == NATCone || t.pn.config.natType == NATCone {
+		if config.PunchPriority&PunchPriorityUDPDisable != 0 {
-		gLog.Println(LvINFO, "try UDP4 Punch")
+			return
-		t.config.linkMode = LinkModeUDPPunch
+		}
-		t.config.isUnderlayServer = 0
+		// try UDPPunch
-		if err = pn.newTunnel(t, tid, isClient); err == nil {
+		for i := 0; i < Cone2ConeUDPPunchMaxRetry; i++ { // when both 2 nats has restrict firewall, simultaneous punching needs to be very precise, it takes a few tries
-			return t, nil
+			if config.peerNatType == NATCone || gConf.Network.natType == NATCone {
-		}
+				gLog.Println(LvINFO, "try UDP4 Punch")
-	}
+				config.linkMode = LinkModeUDPPunch
-	return nil, ErrorHandshake // only ErrorHandshake will try relay
+				config.isUnderlayServer = 0
-}
+				if t, err = pn.newTunnel(config, tid, isClient); err == nil {
-
+					return t, nil
-func (pn *P2PNetwork) newTunnel(t *P2PTunnel, tid uint64, isClient bool) error {
+				}
-	t.initPort()
+			}
-	if isClient {
+			if !(config.peerNatType == NATCone && gConf.Network.natType == NATCone) { // not cone2cone, no more try
-		if err := t.connect(); err != nil {
+				break
-			gLog.Println(LvERROR, "p2pTunnel connect error:", err)
+			}
-			return err
+		}
-		}
+		return
-	} else {
+	}
-		if err := t.listen(); err != nil {
+	funcTCP := func() (t *P2PTunnel, err error) {
-			gLog.Println(LvERROR, "p2pTunnel listen error:", err)
+		if config.PunchPriority&PunchPriorityTCPDisable != 0 {
-			return err
+			return
-		}
+		}
-	}
+		// try TCPPunch
-	// store it when success
+		for i := 0; i < Cone2ConeTCPPunchMaxRetry; i++ { // when both 2 nats has restrict firewall, simultaneous punching needs to be very precise, it takes a few tries
-	gLog.Printf(LvDEBUG, "store tunnel %d", tid)
+			if config.peerNatType == NATCone || gConf.Network.natType == NATCone {
-	pn.allTunnels.Store(tid, t)
+				gLog.Println(LvINFO, "try TCP4 Punch")
-	return nil
+				config.linkMode = LinkModeTCPPunch
-}
+				config.isUnderlayServer = 0
-func (pn *P2PNetwork) init() error {
+				if t, err = pn.newTunnel(config, tid, isClient); err == nil {
-	gLog.Println(LvINFO, "init start")
+					gLog.Println(LvINFO, "TCP4 Punch ok")
-	go func() { //reconnect at least 5s
+					return t, nil
-		pn.wgReconnect.Add(1)
+				}
-		defer pn.wgReconnect.Done()
+			}
-		time.Sleep(NatTestTimeout)
+		}
-	}()
+		return
-	var err error
+	}
-	for {
+	if config.PunchPriority&PunchPriorityTCPFirst != 0 {
-		// detect nat type
+		primaryPunchFunc = funcTCP
-		pn.config.publicIP, pn.config.natType, pn.config.hasIPv4, pn.config.hasUPNPorNATPMP, err = getNATType(pn.config.ServerHost, pn.config.UDPPort1, pn.config.UDPPort2)
+		secondaryPunchFunc = funcUDP
-		// for testcase
+	} else {
-		if strings.Contains(pn.config.Node, "openp2pS2STest") {
+		primaryPunchFunc = funcTCP
-			pn.config.natType = NATSymmetric
+		secondaryPunchFunc = funcUDP
-			pn.config.hasIPv4 = 0
+	}
-			pn.config.hasUPNPorNATPMP = 0
+	if t, err = primaryPunchFunc(); t != nil && err == nil {
-
+		return t, err
-		}
+	}
-		if strings.Contains(pn.config.Node, "openp2pC2CTest") {
+	if t, err = secondaryPunchFunc(); t != nil && err == nil {
-			pn.config.natType = NATCone
+		return t, err
-			pn.config.hasIPv4 = 0
+	}
-			pn.config.hasUPNPorNATPMP = 0
+
-		}
+	// TODO: s2s won't return err
-		if err != nil {
+	return nil, err
-			gLog.Println(LvDEBUG, "detect NAT type error:", err)
+}
-			break
+
-		}
+func (pn *P2PNetwork) newTunnel(config AppConfig, tid uint64, isClient bool) (t *P2PTunnel, err error) {
-		gLog.Println(LvDEBUG, "detect NAT type:", pn.config.natType, " publicIP:", pn.config.publicIP)
+	if isClient { // only client side find existing tunnel
-		gatewayURL := fmt.Sprintf("%s:%d", pn.config.ServerHost, pn.config.ServerPort)
+		if existTunnel := pn.findTunnel(config.PeerNode); existTunnel != nil {
-		uri := "/openp2p/v1/login"
+			return existTunnel, nil
-		config := tls.Config{InsecureSkipVerify: true} // let's encrypt root cert "DST Root CA X3" expired at 2021/09/29. many old system(windows server 2008 etc) will not trust our cert
+		}
-		websocket.DefaultDialer.TLSClientConfig = &config
+	}
-		u := url.URL{Scheme: "wss", Host: gatewayURL, Path: uri}
+
-		q := u.Query()
+	t = &P2PTunnel{pn: pn,
-		q.Add("node", pn.config.Node)
+		config:         config,
-		q.Add("token", fmt.Sprintf("%d", pn.config.Token))
+		id:             tid,
-		q.Add("version", OpenP2PVersion)
+		writeData:      make(chan []byte, WriteDataChanSize),
-		q.Add("nattype", fmt.Sprintf("%d", pn.config.natType))
+		writeDataSmall: make(chan []byte, WriteDataChanSize/30),
-		q.Add("sharebandwidth", fmt.Sprintf("%d", pn.config.ShareBandwidth))
+	}
-		u.RawQuery = q.Encode()
+	t.initPort()
-		var ws *websocket.Conn
+	if isClient {
-		ws, _, err = websocket.DefaultDialer.Dial(u.String(), nil)
+		if err = t.connect(); err != nil {
-		if err != nil {
+			gLog.Println(LvERROR, "p2pTunnel connect error:", err)
-			break
+			return
-		}
+		}
-		pn.online = true
+	} else {
-		pn.conn = ws
+		if err = t.listen(); err != nil {
-		localAddr := strings.Split(ws.LocalAddr().String(), ":")
+			gLog.Println(LvERROR, "p2pTunnel listen error:", err)
-		if len(localAddr) == 2 {
+			return
-			pn.config.localIP = localAddr[0]
+		}
-		} else {
+	}
-			err = errors.New("get local ip failed")
+	// store it when success
-			break
+	gLog.Printf(LvDEBUG, "store tunnel %d", tid)
-		}
+	pn.allTunnels.Store(tid, t)
-		go pn.readLoop()
+	return
-		pn.config.mac = getmac(pn.config.localIP)
+}
-		pn.config.os = getOsName()
+func (pn *P2PNetwork) init() error {
-		go func() {
+	gLog.Println(LvINFO, "P2PNetwork init start")
-			req := ReportBasic{
+	defer gLog.Println(LvINFO, "P2PNetwork init end")
-				Mac:             pn.config.mac,
+	pn.wgReconnect.Add(1)
-				LanIP:           pn.config.localIP,
+	defer pn.wgReconnect.Done()
-				OS:              pn.config.os,
+	var err error
-				HasIPv4:         pn.config.hasIPv4,
+	for {
-				HasUPNPorNATPMP: pn.config.hasUPNPorNATPMP,
+		// detect nat type
-				Version:         OpenP2PVersion,
+		gConf.Network.publicIP, gConf.Network.natType, err = getNATType(gConf.Network.ServerHost, gConf.Network.UDPPort1, gConf.Network.UDPPort2)
-			}
+		if err != nil {
-			rsp := netInfo()
+			gLog.Println(LvDEBUG, "detect NAT type error:", err)
-			gLog.Println(LvDEBUG, "netinfo:", rsp)
+			break
-			if rsp != nil && rsp.Country != "" {
+		}
-				if IsIPv6(rsp.IP.String()) {
+		if gConf.Network.hasIPv4 == 0 && gConf.Network.hasUPNPorNATPMP == 0 { // if already has ipv4 or upnp no need test again
-					pn.config.publicIPv6 = rsp.IP.String()
+			gConf.Network.hasIPv4, gConf.Network.hasUPNPorNATPMP = publicIPTest(gConf.Network.publicIP, gConf.Network.TCPPort)
-				}
+		}
-				req.NetInfo = *rsp
+
-			} else {
+		// for testcase
-				pn.refreshIPv6(true)
+		if strings.Contains(gConf.Network.Node, "openp2pS2STest") {
-			}
+			gConf.Network.natType = NATSymmetric
-			req.IPv6 = pn.config.publicIPv6
+			gConf.Network.hasIPv4 = 0
-			pn.write(MsgReport, MsgReportBasic, &req)
+			gConf.Network.hasUPNPorNATPMP = 0
-		}()
+			gLog.Println(LvINFO, "openp2pS2STest debug")
-		gLog.Println(LvDEBUG, "P2PNetwork init ok")
+
-		break
+		}
-	}
+		if strings.Contains(gConf.Network.Node, "openp2pC2CTest") {
-	if err != nil {
+			gConf.Network.natType = NATCone
-		// init failed, retry
+			gConf.Network.hasIPv4 = 0
-		pn.restartCh <- true
+			gConf.Network.hasUPNPorNATPMP = 0
-		gLog.Println(LvERROR, "P2PNetwork init error:", err)
+			gLog.Println(LvINFO, "openp2pC2CTest debug")
-	}
+		}
-	return err
+
-}
+		if gConf.Network.hasIPv4 == 1 || gConf.Network.hasUPNPorNATPMP == 1 {
-
+			onceV4Listener.Do(func() {
-func (pn *P2PNetwork) handleMessage(t int, msg []byte) {
+				v4l = &v4Listener{port: gConf.Network.TCPPort}
-	head := openP2PHeader{}
+				go v4l.start()
-	err := binary.Read(bytes.NewReader(msg[:openP2PHeaderSize]), binary.LittleEndian, &head)
+			})
-	if err != nil {
+		}
-		gLog.Println(LvERROR, "handleMessage error:", err)
+		gLog.Printf(LvINFO, "hasIPv4:%d, UPNP:%d, NAT type:%d, publicIP:%s", gConf.Network.hasIPv4, gConf.Network.hasUPNPorNATPMP, gConf.Network.natType, gConf.Network.publicIP)
-		return
+		gatewayURL := fmt.Sprintf("%s:%d", gConf.Network.ServerHost, gConf.Network.ServerPort)
-	}
+		uri := "/api/v1/login"
-	switch head.MainType {
+		caCertPool, errCert := x509.SystemCertPool()
-	case MsgLogin:
+		if errCert != nil {
-		// gLog.Println(LevelINFO,string(msg))
+			gLog.Println(LvERROR, "Failed to load system root CAs:", errCert)
-		rsp := LoginRsp{}
+		} else {
-		err = json.Unmarshal(msg[openP2PHeaderSize:], &rsp)
+			caCertPool = x509.NewCertPool()
-		if err != nil {
+		}
-			gLog.Printf(LvERROR, "wrong login response:%s", err)
+		caCertPool.AppendCertsFromPEM([]byte(rootCA))
-			return
+		caCertPool.AppendCertsFromPEM([]byte(ISRGRootX1))
-		}
+		config := tls.Config{
-		if rsp.Error != 0 {
+			RootCAs:            caCertPool,
-			gLog.Printf(LvERROR, "login error:%d, detail:%s", rsp.Error, rsp.Detail)
+			InsecureSkipVerify: false} // let's encrypt root cert "DST Root CA X3" expired at 2021/09/29. many old system(windows server 2008 etc) will not trust our cert
-			pn.running = false
+		websocket.DefaultDialer.TLSClientConfig = &config
-		} else {
+		websocket.DefaultDialer.HandshakeTimeout = ClientAPITimeout
-			pn.serverTs = rsp.Ts
+		u := url.URL{Scheme: "wss", Host: gatewayURL, Path: uri}
-			pn.hbTime = time.Now()
+		q := u.Query()
-			pn.config.Token = rsp.Token
+		q.Add("node", gConf.Network.Node)
-			pn.config.User = rsp.User
+		q.Add("token", fmt.Sprintf("%d", gConf.Network.Token))
-			gConf.setToken(rsp.Token)
+		q.Add("version", OpenP2PVersion)
-			gConf.setUser(rsp.User)
+		q.Add("nattype", fmt.Sprintf("%d", gConf.Network.natType))
-			if len(rsp.Node) >= MinNodeNameLen {
+		q.Add("sharebandwidth", fmt.Sprintf("%d", gConf.Network.ShareBandwidth))
-				gConf.setNode(rsp.Node)
+		u.RawQuery = q.Encode()
-			}
+		var ws *websocket.Conn
-			gConf.save()
+		ws, _, err = websocket.DefaultDialer.Dial(u.String(), nil)
-			pn.localTs = time.Now().Unix()
+		if err != nil {
-			gLog.Printf(LvINFO, "login ok. user=%s,node=%s,Server ts=%d, local ts=%d", rsp.User, rsp.Node, rsp.Ts, pn.localTs)
+			gLog.Println(LvERROR, "Dial error:", err)
-		}
+			break
-	case MsgHeartbeat:
+		}
-		gLog.Printf(LvDEBUG, "P2PNetwork heartbeat ok")
+		pn.running = true
-		pn.hbTime = time.Now()
+		pn.online = true
-	case MsgPush:
+		pn.conn = ws
-		handlePush(pn, head.SubType, msg)
+		localAddr := strings.Split(ws.LocalAddr().String(), ":")
-	default:
+		if len(localAddr) == 2 {
-		pn.msgMapMtx.Lock()
+			gConf.Network.localIP = localAddr[0]
-		ch := pn.msgMap[0]
+		} else {
-		pn.msgMapMtx.Unlock()
+			err = errors.New("get local ip failed")
-		ch <- msg
+			break
-		return
+		}
-	}
+		go pn.readLoop()
-}
+		gConf.Network.mac = getmac(gConf.Network.localIP)
-
+		gConf.Network.os = getOsName()
-func (pn *P2PNetwork) readLoop() {
+		go func() {
-	gLog.Printf(LvDEBUG, "P2PNetwork readLoop start")
+			req := ReportBasic{
-	pn.wgReconnect.Add(1)
+				Mac:             gConf.Network.mac,
-	defer pn.wgReconnect.Done()
+				LanIP:           gConf.Network.localIP,
-	for pn.running {
+				OS:              gConf.Network.os,
-		pn.conn.SetReadDeadline(time.Now().Add(NetworkHeartbeatTime + 10*time.Second))
+				HasIPv4:         gConf.Network.hasIPv4,
-		t, msg, err := pn.conn.ReadMessage()
+				HasUPNPorNATPMP: gConf.Network.hasUPNPorNATPMP,
-		if err != nil {
+				Version:         OpenP2PVersion,
-			gLog.Printf(LvERROR, "P2PNetwork read error:%s", err)
+			}
-			pn.conn.Close()
+			rsp := netInfo()
-			pn.restartCh <- true
+			gLog.Println(LvDEBUG, "netinfo:", rsp)
-			break
+			if rsp != nil && rsp.Country != "" {
-		}
+				if IsIPv6(rsp.IP.String()) {
-		pn.handleMessage(t, msg)
+					gConf.setIPv6(rsp.IP.String())
-	}
+				}
-	gLog.Printf(LvDEBUG, "P2PNetwork readLoop end")
+				req.NetInfo = *rsp
-}
+			} else {
-
+				pn.refreshIPv6()
-func (pn *P2PNetwork) write(mainType uint16, subType uint16, packet interface{}) error {
+			}
-	if !pn.online {
+			req.IPv6 = gConf.IPv6()
-		return errors.New("P2P network offline")
+			pn.write(MsgReport, MsgReportBasic, &req)
-	}
+		}()
-	msg, err := newMessage(mainType, subType, packet)
+		go pn.autorunApp()
-	if err != nil {
+		pn.write(MsgSDWAN, MsgSDWANInfoReq, nil)
-		return err
+		gLog.Println(LvDEBUG, "P2PNetwork init ok")
-	}
+		break
-	pn.writeMtx.Lock()
+	}
-	defer pn.writeMtx.Unlock()
+	if err != nil {
-	if err = pn.conn.WriteMessage(websocket.BinaryMessage, msg); err != nil {
+		// init failed, retry
-		gLog.Printf(LvERROR, "write msgType %d,%d error:%s", mainType, subType, err)
+		pn.close()
-		pn.conn.Close()
+		gLog.Println(LvERROR, "P2PNetwork init error:", err)
-	}
+	}
-	return err
+	return err
-}
+}
-
+
-func (pn *P2PNetwork) relay(to uint64, body []byte) error {
+func (pn *P2PNetwork) handleMessage(msg []byte) {
-	gLog.Printf(LvDEBUG, "relay data to %d", to)
+	head := openP2PHeader{}
-	i, ok := pn.allTunnels.Load(to)
+	err := binary.Read(bytes.NewReader(msg[:openP2PHeaderSize]), binary.LittleEndian, &head)
-	if !ok {
+	if err != nil {
-		return nil
+		gLog.Println(LvERROR, "handleMessage error:", err)
-	}
+		return
-	tunnel := i.(*P2PTunnel)
+	}
-	if tunnel.config.shareBandwidth > 0 {
+	switch head.MainType {
-		pn.limiter.Add(len(body))
+	case MsgLogin:
-	}
+		// gLog.Println(LevelINFO,string(msg))
-	tunnel.conn.WriteBuffer(body)
+		rsp := LoginRsp{}
-	return nil
+		if err = json.Unmarshal(msg[openP2PHeaderSize:], &rsp); err != nil {
-}
+			gLog.Printf(LvERROR, "wrong %v:%s", reflect.TypeOf(rsp), err)
-
+			return
-func (pn *P2PNetwork) push(to string, subType uint16, packet interface{}) error {
+		}
-	gLog.Printf(LvDEBUG, "push msgType %d to %s", subType, to)
+		if rsp.Error != 0 {
-	if !pn.online {
+			gLog.Printf(LvERROR, "login error:%d, detail:%s", rsp.Error, rsp.Detail)
-		return errors.New("client offline")
+			pn.running = false
-	}
+		} else {
-	pushHead := PushHeader{}
+			gConf.setToken(rsp.Token)
-	pushHead.From = nodeNameToID(pn.config.Node)
+			gConf.setUser(rsp.User)
-	pushHead.To = nodeNameToID(to)
+			if len(rsp.Node) >= MinNodeNameLen {
-	pushHeadBuf := new(bytes.Buffer)
+				gConf.setNode(rsp.Node)
-	err := binary.Write(pushHeadBuf, binary.LittleEndian, pushHead)
+			}
-	if err != nil {
+			if rsp.LoginMaxDelay > 0 {
-		return err
+				pn.loginMaxDelaySeconds = rsp.LoginMaxDelay
-	}
+			}
-	data, err := json.Marshal(packet)
+			gLog.Printf(LvINFO, "login ok. user=%s,node=%s", rsp.User, rsp.Node)
-	if err != nil {
+		}
-		return err
+	case MsgHeartbeat:
-	}
+		gLog.Printf(LvDev, "P2PNetwork heartbeat ok")
-	// gLog.Println(LevelINFO,"write packet:", string(data))
+		pn.hbTime = time.Now()
-	pushMsg := append(encodeHeader(MsgPush, subType, uint32(len(data)+PushHeaderSize)), pushHeadBuf.Bytes()...)
+		rtt := pn.hbTime.UnixNano() - pn.t1
-	pushMsg = append(pushMsg, data...)
+		if rtt > int64(PunchTsDelay) || (pn.preRtt > 0 && rtt > pn.preRtt*5) {
-	pn.writeMtx.Lock()
+			gLog.Printf(LvINFO, "rtt=%d too large ignore", rtt)
-	defer pn.writeMtx.Unlock()
+			return // invalid hb rsp
-	if err = pn.conn.WriteMessage(websocket.BinaryMessage, pushMsg); err != nil {
+		}
-		gLog.Printf(LvERROR, "push to %s error:%s", to, err)
+		pn.preRtt = rtt
-		pn.conn.Close()
+		t2 := int64(binary.LittleEndian.Uint64(msg[openP2PHeaderSize : openP2PHeaderSize+8]))
-	}
+		thisdt := pn.t1 + rtt/2 - t2
-	return err
+		newdt := thisdt
-}
+		if pn.dt != 0 {
-
+			ddt := thisdt - pn.dt
-func (pn *P2PNetwork) read(node string, mainType uint16, subType uint16, timeout time.Duration) (head *openP2PHeader, body []byte) {
+			pn.ddt = ddt
-	var nodeID uint64
+			if pn.ddtma == 0 {
-	if node == "" {
+				pn.ddtma = pn.ddt
-		nodeID = 0
+			} else {
-	} else {
+				pn.ddtma = int64(float64(pn.ddtma)*(1-ma10) + float64(pn.ddt)*ma10) // avoid int64 overflow
-		nodeID = nodeNameToID(node)
+				newdt = pn.dt + pn.ddtma
-	}
+			}
-	pn.msgMapMtx.Lock()
+		}
-	ch := pn.msgMap[nodeID]
+		pn.dt = newdt
-	pn.msgMapMtx.Unlock()
+		gLog.Printf(LvDEBUG, "synctime thisdt=%dms dt=%dms ddt=%dns ddtma=%dns rtt=%dms ", thisdt/int64(time.Millisecond), pn.dt/int64(time.Millisecond), pn.ddt, pn.ddtma, rtt/int64(time.Millisecond))
-	for {
+	case MsgPush:
-		select {
+		handlePush(head.SubType, msg)
-		case <-time.After(timeout):
+	case MsgSDWAN:
-			gLog.Printf(LvERROR, "wait msg%d:%d timeout", mainType, subType)
+		handleSDWAN(head.SubType, msg)
-			return
+	default:
-		case msg := <-ch:
+		i, ok := pn.msgMap.Load(uint64(0))
-			head = &openP2PHeader{}
+		if ok {
-			err := binary.Read(bytes.NewReader(msg[:openP2PHeaderSize]), binary.LittleEndian, head)
+			ch := i.(chan msgCtx)
-			if err != nil {
+			ch <- msgCtx{data: msg, ts: time.Now()}
-				gLog.Println(LvERROR, "read msg error:", err)
+		}
-				break
+
-			}
+		return
-			if head.MainType != mainType || head.SubType != subType {
+	}
-				continue
+}
-			}
+
-			if mainType == MsgPush {
+func (pn *P2PNetwork) readLoop() {
-				body = msg[openP2PHeaderSize+PushHeaderSize:]
+	gLog.Printf(LvDEBUG, "P2PNetwork readLoop start")
-			} else {
+	pn.wgReconnect.Add(1)
-				body = msg[openP2PHeaderSize:]
+	defer pn.wgReconnect.Done()
-			}
+	for pn.running {
-			return
+		pn.conn.SetReadDeadline(time.Now().Add(NetworkHeartbeatTime + 10*time.Second))
-		}
+		_, msg, err := pn.conn.ReadMessage()
-	}
+		if err != nil {
-}
+			gLog.Printf(LvERROR, "P2PNetwork read error:%s", err)
-
+			pn.close()
-func (pn *P2PNetwork) updateAppHeartbeat(appID uint64) {
+			break
-	pn.apps.Range(func(id, i interface{}) bool {
+		}
-		app := i.(*p2pApp)
+		pn.handleMessage(msg)
-		if app.id != appID {
+	}
-			return true
+	gLog.Printf(LvDEBUG, "P2PNetwork readLoop end")
-		}
+}
-		app.updateHeartbeat()
+
-		return false
+func (pn *P2PNetwork) write(mainType uint16, subType uint16, packet interface{}) error {
-	})
+	if !pn.online {
-}
+		return errors.New("P2P network offline")
-
+	}
-func (pn *P2PNetwork) refreshIPv6(force bool) {
+	msg, err := newMessage(mainType, subType, packet)
-	if !force && !IsIPv6(pn.config.publicIPv6) { // not support ipv6, not refresh
+	if err != nil {
-		return
+		return err
-	}
+	}
-	client := &http.Client{Timeout: time.Second * 10}
+	pn.writeMtx.Lock()
-	r, err := client.Get("http://6.ipw.cn")
+	defer pn.writeMtx.Unlock()
-	if err != nil {
+	if err = pn.conn.WriteMessage(websocket.BinaryMessage, msg); err != nil {
-		gLog.Println(LvDEBUG, "refreshIPv6 error:", err)
+		gLog.Printf(LvERROR, "write msgType %d,%d error:%s", mainType, subType, err)
-		return
+		pn.close()
-	}
+	}
-	defer r.Body.Close()
+	return err
-	buf := make([]byte, 1024)
+}
-	n, err := r.Body.Read(buf)
+
-	if n <= 0 {
+func (pn *P2PNetwork) relay(to uint64, body []byte) error {
-		gLog.Println(LvINFO, "refreshIPv6 error:", err, n)
+	i, ok := pn.allTunnels.Load(to)
-		return
+	if !ok {
-	}
+		return ErrRelayTunnelNotFound
-	pn.config.publicIPv6 = string(buf[:n])
+	}
-}
+	tunnel := i.(*P2PTunnel)
+	if tunnel.config.shareBandwidth > 0 {
+		pn.limiter.Add(len(body), true)
+	}
+	var err error
+	if err = tunnel.conn.WriteBuffer(body); err != nil {
+		gLog.Printf(LvERROR, "relay to %d len=%d error:%s", to, len(body), err)
+	}
+	return err
+}
+
+func (pn *P2PNetwork) push(to string, subType uint16, packet interface{}) error {
+	// gLog.Printf(LvDEBUG, "push msgType %d to %s", subType, to)
+	if !pn.online {
+		return errors.New("client offline")
+	}
+	pushHead := PushHeader{}
+	pushHead.From = gConf.nodeID()
+	pushHead.To = NodeNameToID(to)
+	pushHeadBuf := new(bytes.Buffer)
+	err := binary.Write(pushHeadBuf, binary.LittleEndian, pushHead)
+	if err != nil {
+		return err
+	}
+	data, err := json.Marshal(packet)
+	if err != nil {
+		return err
+	}
+	// gLog.Println(LevelINFO,"write packet:", string(data))
+	pushMsg := append(encodeHeader(MsgPush, subType, uint32(len(data)+PushHeaderSize)), pushHeadBuf.Bytes()...)
+	pushMsg = append(pushMsg, data...)
+	pn.writeMtx.Lock()
+	defer pn.writeMtx.Unlock()
+	if err = pn.conn.WriteMessage(websocket.BinaryMessage, pushMsg); err != nil {
+		gLog.Printf(LvERROR, "push to %s error:%s", to, err)
+		pn.close()
+	}
+	return err
+}
+
+func (pn *P2PNetwork) close() {
+	if pn.running {
+		if pn.conn != nil {
+			pn.conn.Close()
+		}
+		pn.running = false
+	}
+	select {
+	case pn.restartCh <- true:
+	default:
+	}
+}
+
+func (pn *P2PNetwork) read(node string, mainType uint16, subType uint16, timeout time.Duration) (head *openP2PHeader, body []byte) {
+	var nodeID uint64
+	if node == "" {
+		nodeID = 0
+	} else {
+		nodeID = NodeNameToID(node)
+	}
+	i, ok := pn.msgMap.Load(nodeID)
+	if !ok {
+		gLog.Printf(LvERROR, "read msg error: %s not found", node)
+		return
+	}
+	ch := i.(chan msgCtx)
+	for {
+		select {
+		case <-time.After(timeout):
+			gLog.Printf(LvERROR, "read msg error %d:%d timeout", mainType, subType)
+			return
+		case msg := <-ch:
+			head = &openP2PHeader{}
+			err := binary.Read(bytes.NewReader(msg.data[:openP2PHeaderSize]), binary.LittleEndian, head)
+			if err != nil {
+				gLog.Println(LvERROR, "read msg error:", err)
+				break
+			}
+			if time.Since(msg.ts) > ReadMsgTimeout {
+				gLog.Printf(LvDEBUG, "read msg error expired %d:%d", head.MainType, head.SubType)
+				continue
+			}
+			if head.MainType != mainType || head.SubType != subType {
+				gLog.Printf(LvDEBUG, "read msg error type %d:%d, requeue it", head.MainType, head.SubType)
+				ch <- msg
+				time.Sleep(time.Second)
+				continue
+			}
+			if mainType == MsgPush {
+				body = msg.data[openP2PHeaderSize+PushHeaderSize:]
+			} else {
+				body = msg.data[openP2PHeaderSize:]
+			}
+			return
+		}
+	}
+}
+
+func (pn *P2PNetwork) updateAppHeartbeat(appID uint64) {
+	pn.apps.Range(func(id, i interface{}) bool {
+		app := i.(*p2pApp)
+		if app.id == appID {
+			app.updateHeartbeat()
+		}
+		return true
+	})
+}
+
+// ipv6 will expired need to refresh.
+func (pn *P2PNetwork) refreshIPv6() {
+	for i := 0; i < 2; i++ {
+		client := &http.Client{Timeout: time.Second * 10}
+		r, err := client.Get("http://ipv6.ddnspod.com/")
+		if err != nil {
+			gLog.Println(LvDEBUG, "refreshIPv6 error:", err)
+			continue
+		}
+		defer r.Body.Close()
+		buf := make([]byte, 1024)
+		n, err := r.Body.Read(buf)
+		if n <= 0 {
+			gLog.Println(LvINFO, "refreshIPv6 error:", err, n)
+			continue
+		}
+		if IsIPv6(string(buf[:n])) {
+			gConf.setIPv6(string(buf[:n]))
+		}
+		break
+	}
+
+}
+
+func (pn *P2PNetwork) requestPeerInfo(config *AppConfig) error {
+	// request peer info
+	// TODO: multi-thread issue
+	pn.reqGatewayMtx.Lock()
+	pn.write(MsgQuery, MsgQueryPeerInfoReq, &QueryPeerInfoReq{config.peerToken, config.PeerNode})
+	head, body := pn.read("", MsgQuery, MsgQueryPeerInfoRsp, ClientAPITimeout)
+	pn.reqGatewayMtx.Unlock()
+	if head == nil {
+		gLog.Println(LvERROR, "requestPeerInfo error")
+		return ErrNetwork // network error, should not be ErrPeerOffline
+	}
+	rsp := QueryPeerInfoRsp{}
+	if err := json.Unmarshal(body, &rsp); err != nil {
+		return ErrMsgFormat
+	}
+	if rsp.Online == 0 {
+		return ErrPeerOffline
+	}
+	if compareVersion(rsp.Version, LeastSupportVersion) < 0 {
+		return ErrVersionNotCompatible
+	}
+	config.peerVersion = rsp.Version
+	config.peerLanIP = rsp.LanIP
+	config.hasIPv4 = rsp.HasIPv4
+	config.peerIP = rsp.IPv4
+	config.peerIPv6 = rsp.IPv6
+	config.hasUPNPorNATPMP = rsp.HasUPNPorNATPMP
+	config.peerNatType = rsp.NatType
+	///
+	return nil
+}
+
+func (pn *P2PNetwork) StartSDWAN() {
+	// request peer info
+	pn.sdwan = &p2pSDWAN{}
+}
+
+func (pn *P2PNetwork) ConnectNode(node string) error {
+	if gConf.nodeID() < NodeNameToID(node) {
+		return errors.New("only the bigger nodeid connect")
+	}
+	peerNodeID := fmt.Sprintf("%d", NodeNameToID(node))
+	config := AppConfig{Enabled: 1}
+	config.AppName = peerNodeID
+	config.SrcPort = 0
+	config.PeerNode = node
+	sdwan := gConf.getSDWAN()
+	config.PunchPriority = int(sdwan.PunchPriority)
+	if node != sdwan.CentralNode && gConf.Network.Node != sdwan.CentralNode { // neither is centralnode
+		config.RelayNode = sdwan.CentralNode
+		config.ForceRelay = int(sdwan.ForceRelay)
+		if sdwan.Mode == SDWANModeCentral {
+			config.ForceRelay = 1
+		}
+	}
+
+	gConf.add(config, true)
+	return nil
+}
+
+func (pn *P2PNetwork) WriteNode(nodeID uint64, buff []byte) error {
+	i, ok := pn.apps.Load(nodeID)
+	if !ok {
+		return errors.New("peer not found")
+	}
+	var err error
+	app := i.(*p2pApp)
+	if app.Tunnel() == nil {
+		return errors.New("peer tunnel nil")
+	}
+	// TODO: move to app.write
+	gLog.Printf(LvDev, "%d tunnel write node data bodylen=%d, relay=%t", app.Tunnel().id, len(buff), !app.isDirect())
+	if app.isDirect() { // direct
+		app.Tunnel().asyncWriteNodeData(MsgP2P, MsgNodeData, buff)
+	} else { // relay
+		fromNodeIDHead := new(bytes.Buffer)
+		binary.Write(fromNodeIDHead, binary.LittleEndian, gConf.nodeID())
+		all := app.RelayHead().Bytes()
+		all = append(all, encodeHeader(MsgP2P, MsgRelayNodeData, uint32(len(buff)+overlayHeaderSize))...)
+		all = append(all, fromNodeIDHead.Bytes()...)
+		all = append(all, buff...)
+		app.Tunnel().asyncWriteNodeData(MsgP2P, MsgRelayData, all)
+	}
+
+	return err
+}
+
+func (pn *P2PNetwork) WriteBroadcast(buff []byte) error {
+	///
+	pn.apps.Range(func(id, i interface{}) bool {
+		// newDestIP := net.ParseIP("10.2.3.2")
+		// copy(buff[16:20], newDestIP.To4())
+		// binary.BigEndian.PutUint16(buff[10:12], 0) // set checksum=0 for calc checksum
+		// ipChecksum := calculateChecksum(buff[0:20])
+		// binary.BigEndian.PutUint16(buff[10:12], ipChecksum)
+		// binary.BigEndian.PutUint16(buff[26:28], 0x082e)
+		app := i.(*p2pApp)
+		if app.Tunnel() == nil {
+			return true
+		}
+		if app.config.SrcPort != 0 { // normal portmap app
+			return true
+		}
+		if app.config.peerIP == gConf.Network.publicIP { // mostly in a lan
+			return true
+		}
+		if app.isDirect() { // direct
+			app.Tunnel().conn.WriteBytes(MsgP2P, MsgNodeData, buff)
+		} else { // relay
+			fromNodeIDHead := new(bytes.Buffer)
+			binary.Write(fromNodeIDHead, binary.LittleEndian, gConf.nodeID())
+			all := app.RelayHead().Bytes()
+			all = append(all, encodeHeader(MsgP2P, MsgRelayNodeData, uint32(len(buff)+overlayHeaderSize))...)
+			all = append(all, fromNodeIDHead.Bytes()...)
+			all = append(all, buff...)
+			app.Tunnel().conn.WriteBytes(MsgP2P, MsgRelayData, all)
+		}
+		return true
+	})
+	return nil
+}
+
+func (pn *P2PNetwork) ReadNode(tm time.Duration) *NodeData {
+	select {
+	case nd := <-pn.nodeData:
+		return nd
+	case <-time.After(tm):
+	}
+	return nil
+}

core/p2ptunnel.go

@@ -1,624 +1,806 @@
-package openp2p
+package openp2p
-
+
-import (
+import (
-	"bytes"
+	"bytes"
-	"encoding/binary"
+	"encoding/binary"
-	"encoding/json"
+	"encoding/json"
-	"errors"
+	"errors"
-	"fmt"
+	"fmt"
-	"math/rand"
+	"math/rand"
-	"net"
+	"net"
-	"sync"
+	"reflect"
-	"time"
+	"sync"
-)
+	"sync/atomic"
-
+	"time"
-type P2PTunnel struct {
+)
-	pn            *P2PNetwork
+
-	conn          underlay
+const WriteDataChanSize int = 3000
-	hbTime        time.Time
+
-	hbMtx         sync.Mutex
+var buildTunnelMtx sync.Mutex
-	hbTimeRelay   time.Time
+
-	config        AppConfig
+type P2PTunnel struct {
-	la            *net.UDPAddr // local hole address
+	pn             *P2PNetwork
-	ra            *net.UDPAddr // remote hole address
+	conn           underlay
-	overlayConns  sync.Map     // both TCP and UDP
+	hbTime         time.Time
-	id            uint64
+	hbMtx          sync.Mutex
-	running       bool
+	config         AppConfig
-	runMtx        sync.Mutex
+	la             *net.UDPAddr // local hole address
-	tunnelServer  bool // different from underlayServer
+	ra             *net.UDPAddr // remote hole address
-	coneLocalPort int
+	overlayConns   sync.Map     // both TCP and UDP
-	coneNatPort   int
+	id             uint64       // client side alloc rand.uint64 = server side
-	linkModeWeb   string // use config.linkmode
+	running        bool
-}
+	runMtx         sync.Mutex
-
+	tunnelServer   bool // different from underlayServer
-func (t *P2PTunnel) requestPeerInfo() error {
+	coneLocalPort  int
-	// request peer info
+	coneNatPort    int
-	t.pn.write(MsgQuery, MsgQueryPeerInfoReq, &QueryPeerInfoReq{t.config.peerToken, t.config.PeerNode})
+	linkModeWeb    string // use config.linkmode
-	head, body := t.pn.read("", MsgQuery, MsgQueryPeerInfoRsp, time.Second*10)
+	punchTs        uint64
-	if head == nil {
+	writeData      chan []byte
-		return ErrPeerOffline
+	writeDataSmall chan []byte
-	}
+}
-	rsp := QueryPeerInfoRsp{}
+
-	err := json.Unmarshal(body, &rsp)
+func (t *P2PTunnel) initPort() {
-	if err != nil {
+	t.running = true
-		gLog.Printf(LvERROR, "wrong QueryPeerInfoRsp:%s", err)
+	localPort := int(rand.Uint32()%15000 + 50000) // if the process has bug, will add many upnp port. use specify p2p port by param
-		return ErrMsgFormat
+	if t.config.linkMode == LinkModeTCP6 || t.config.linkMode == LinkModeTCP4 || t.config.linkMode == LinkModeIntranet {
-	}
+		t.coneLocalPort = gConf.Network.TCPPort
-	if rsp.Online == 0 {
+		t.coneNatPort = gConf.Network.TCPPort // symmetric doesn't need coneNatPort
-		return ErrPeerOffline
+	}
-	}
+	if t.config.linkMode == LinkModeUDPPunch {
-	if compareVersion(rsp.Version, LeastSupportVersion) == LESS {
+		// prepare one random cone hole manually
-		return ErrVersionNotCompatible
+		_, natPort, _ := natTest(gConf.Network.ServerHost, gConf.Network.UDPPort1, localPort)
-	}
+		t.coneLocalPort = localPort
-	t.config.peerVersion = rsp.Version
+		t.coneNatPort = natPort
-	t.config.hasIPv4 = rsp.HasIPv4
+	}
-	t.config.peerIP = rsp.IPv4
+	if t.config.linkMode == LinkModeTCPPunch {
-	t.config.peerIPv6 = rsp.IPv6
+		// prepare one random cone hole by system automatically
-	t.config.hasUPNPorNATPMP = rsp.HasUPNPorNATPMP
+		_, natPort, localPort2 := natTCP(gConf.Network.ServerHost, IfconfigPort1)
-	t.config.peerNatType = rsp.NatType
+		t.coneLocalPort = localPort2
-	///
+		t.coneNatPort = natPort
-	return nil
+	}
-}
+	t.la = &net.UDPAddr{IP: net.ParseIP(gConf.Network.localIP), Port: t.coneLocalPort}
-func (t *P2PTunnel) initPort() {
+	gLog.Printf(LvDEBUG, "prepare punching port %d:%d", t.coneLocalPort, t.coneNatPort)
-	t.running = true
+}
-	t.hbMtx.Lock()
+
-	t.hbTime = time.Now()
+func (t *P2PTunnel) connect() error {
-	t.hbMtx.Unlock()
+	gLog.Printf(LvDEBUG, "start p2pTunnel to %s ", t.config.LogPeerNode())
-	t.hbTimeRelay = time.Now().Add(time.Second * 600) // TODO: test fake time
+	t.tunnelServer = false
-	localPort := int(rand.Uint32()%15000 + 50000)     // if the process has bug, will add many upnp port. use specify p2p port by param
+	appKey := uint64(0)
-	if t.config.linkMode == LinkModeTCP6 {
+	req := PushConnectReq{
-		t.pn.refreshIPv6(false)
+		Token:            t.config.peerToken,
-	}
+		From:             gConf.Network.Node,
-	if t.config.linkMode == LinkModeTCP6 || t.config.linkMode == LinkModeTCP4 {
+		FromIP:           gConf.Network.publicIP,
-		t.coneLocalPort = t.pn.config.TCPPort
+		ConeNatPort:      t.coneNatPort,
-		t.coneNatPort = t.pn.config.TCPPort // symmetric doesn't need coneNatPort
+		NatType:          gConf.Network.natType,
-	}
+		HasIPv4:          gConf.Network.hasIPv4,
-	if t.config.linkMode == LinkModeUDPPunch {
+		IPv6:             gConf.IPv6(),
-		// prepare one random cone hole
+		HasUPNPorNATPMP:  gConf.Network.hasUPNPorNATPMP,
-		_, natPort, _ := natTest(t.pn.config.ServerHost, t.pn.config.UDPPort1, localPort)
+		ID:               t.id,
-		t.coneLocalPort = localPort
+		AppKey:           appKey,
-		t.coneNatPort = natPort
+		Version:          OpenP2PVersion,
-	}
+		LinkMode:         t.config.linkMode,
-	if t.config.linkMode == LinkModeTCPPunch {
+		IsUnderlayServer: t.config.isUnderlayServer ^ 1, // peer
-		// prepare one random cone hole
+		UnderlayProtocol: t.config.UnderlayProtocol,
-		_, natPort := natTCP(t.pn.config.ServerHost, IfconfigPort1, localPort)
+	}
-		t.coneLocalPort = localPort
+	if req.Token == 0 { // no relay token
-		t.coneNatPort = natPort
+		req.Token = gConf.Network.Token
-	}
+	}
-	t.la = &net.UDPAddr{IP: net.ParseIP(t.pn.config.localIP), Port: t.coneLocalPort}
+	t.pn.push(t.config.PeerNode, MsgPushConnectReq, req)
-	gLog.Printf(LvDEBUG, "prepare punching port %d:%d", t.coneLocalPort, t.coneNatPort)
+	head, body := t.pn.read(t.config.PeerNode, MsgPush, MsgPushConnectRsp, UnderlayConnectTimeout*3)
-}
+	if head == nil {
-
+		return errors.New("connect error")
-func (t *P2PTunnel) connect() error {
+	}
-	gLog.Printf(LvDEBUG, "start p2pTunnel to %s ", t.config.PeerNode)
+	rsp := PushConnectRsp{}
-	t.tunnelServer = false
+	if err := json.Unmarshal(body, &rsp); err != nil {
-	appKey := uint64(0)
+		gLog.Printf(LvERROR, "wrong %v:%s", reflect.TypeOf(rsp), err)
-	req := PushConnectReq{
+		return err
-		Token:            t.config.peerToken,
+	}
-		From:             t.pn.config.Node,
+	// gLog.Println(LevelINFO, rsp)
-		FromIP:           t.pn.config.publicIP,
+	if rsp.Error != 0 {
-		ConeNatPort:      t.coneNatPort,
+		return errors.New(rsp.Detail)
-		NatType:          t.pn.config.natType,
+	}
-		HasIPv4:          t.pn.config.hasIPv4,
+	t.config.peerNatType = rsp.NatType
-		IPv6:             t.pn.config.publicIPv6,
+	t.config.hasIPv4 = rsp.HasIPv4
-		HasUPNPorNATPMP:  t.pn.config.hasUPNPorNATPMP,
+	t.config.peerIPv6 = rsp.IPv6
-		ID:               t.id,
+	t.config.hasUPNPorNATPMP = rsp.HasUPNPorNATPMP
-		AppKey:           appKey,
+	t.config.peerVersion = rsp.Version
-		Version:          OpenP2PVersion,
+	t.config.peerConeNatPort = rsp.ConeNatPort
-		LinkMode:         t.config.linkMode,
+	t.config.peerIP = rsp.FromIP
-		IsUnderlayServer: t.config.isUnderlayServer ^ 1, // peer
+	t.punchTs = rsp.PunchTs
-	}
+	err := t.start()
-	if req.Token == 0 { // no relay token
+	if err != nil {
-		req.Token = t.pn.config.Token
+		gLog.Println(LvERROR, "handshake error:", err)
-	}
+	}
-	t.pn.push(t.config.PeerNode, MsgPushConnectReq, req)
+	return err
-	head, body := t.pn.read(t.config.PeerNode, MsgPush, MsgPushConnectRsp, time.Second*10)
+}
-	if head == nil {
+
-		return errors.New("connect error")
+func (t *P2PTunnel) isRuning() bool {
-	}
+	t.runMtx.Lock()
-	rsp := PushConnectRsp{}
+	defer t.runMtx.Unlock()
-	err := json.Unmarshal(body, &rsp)
+	return t.running
-	if err != nil {
+}
-		gLog.Printf(LvERROR, "wrong MsgPushConnectRsp:%s", err)
+
-		return err
+func (t *P2PTunnel) setRun(running bool) {
-	}
+	t.runMtx.Lock()
-	// gLog.Println(LevelINFO, rsp)
+	defer t.runMtx.Unlock()
-	if rsp.Error != 0 {
+	t.running = running
-		return errors.New(rsp.Detail)
+}
-	}
+
-	t.config.peerNatType = rsp.NatType
+func (t *P2PTunnel) isActive() bool {
-	t.config.hasIPv4 = rsp.HasIPv4
+	if !t.isRuning() || t.conn == nil {
-	t.config.peerIPv6 = rsp.IPv6
+		return false
-	t.config.hasUPNPorNATPMP = rsp.HasUPNPorNATPMP
+	}
-	t.config.peerVersion = rsp.Version
+	t.hbMtx.Lock()
-	t.config.peerConeNatPort = rsp.ConeNatPort
+	defer t.hbMtx.Unlock()
-	t.config.peerIP = rsp.FromIP
+	res := time.Now().Before(t.hbTime.Add(TunnelHeartbeatTime * 2))
-	err = t.start()
+	if !res {
-	if err != nil {
+		gLog.Printf(LvDEBUG, "%d tunnel isActive false", t.id)
-		gLog.Println(LvERROR, "handshake error:", err)
+	}
-		err = ErrorHandshake
+	return res
-	}
+}
-	return err
+
-}
+func (t *P2PTunnel) checkActive() bool {
-
+	if !t.isActive() {
-func (t *P2PTunnel) isRuning() bool {
+		return false
-	t.runMtx.Lock()
+	}
-	defer t.runMtx.Unlock()
+	hbt := time.Now()
-	return t.running
+	t.conn.WriteBytes(MsgP2P, MsgTunnelHeartbeat, nil)
-}
+	isActive := false
-
+	// wait at most 5s
-func (t *P2PTunnel) setRun(running bool) {
+	for i := 0; i < 50 && !isActive; i++ {
-	t.runMtx.Lock()
+		t.hbMtx.Lock()
-	defer t.runMtx.Unlock()
+		if t.hbTime.After(hbt) {
-	t.running = running
+			isActive = true
-}
+		}
-
+		t.hbMtx.Unlock()
-func (t *P2PTunnel) isActive() bool {
+		time.Sleep(time.Millisecond * 100)
-	if !t.isRuning() {
+	}
-		return false
+	gLog.Printf(LvINFO, "checkActive %t. hbtime=%d", isActive, t.hbTime)
-	}
+	return isActive
-	t.hbMtx.Lock()
+}
-	defer t.hbMtx.Unlock()
+
-	return time.Now().Before(t.hbTime.Add(TunnelIdleTimeout))
+// call when user delete tunnel
-}
+func (t *P2PTunnel) close() {
-
+	t.pn.NotifyTunnelClose(t)
-func (t *P2PTunnel) checkActive() bool {
+	if !t.running {
-	hbt := time.Now()
+		return
-	t.hbMtx.Lock()
+	}
-	if t.hbTime.Before(time.Now().Add(-TunnelHeartbeatTime)) {
+	t.setRun(false)
-		t.hbMtx.Unlock()
+	if t.conn != nil {
-		return false
+		t.conn.Close()
-	}
+	}
-	t.hbMtx.Unlock()
+	t.pn.allTunnels.Delete(t.id)
-	// hbtime within TunnelHeartbeatTime, check it now
+	gLog.Printf(LvINFO, "%d p2ptunnel close %s ", t.id, t.config.LogPeerNode())
-	t.conn.WriteBytes(MsgP2P, MsgTunnelHeartbeat, nil)
+}
-	isActive := false
+
-	// wait at most 5s
+func (t *P2PTunnel) start() error {
-	for i := 0; i < 50 && !isActive; i++ {
+	if t.config.linkMode == LinkModeUDPPunch {
-		t.hbMtx.Lock()
+		if err := t.handshake(); err != nil {
-		if t.hbTime.After(hbt) {
+			return err
-			isActive = true
+		}
-		}
+	}
-		t.hbMtx.Unlock()
+	err := t.connectUnderlay()
-		time.Sleep(time.Millisecond * 100)
+	if err != nil {
-	}
+		gLog.Println(LvERROR, err)
-	return isActive
+		return err
-}
+	}
-
+	return nil
-// call when user delete tunnel
+}
-func (t *P2PTunnel) close() {
+
-	t.setRun(false)
+func (t *P2PTunnel) handshake() error {
-	t.pn.allTunnels.Delete(t.id)
+	if t.config.peerConeNatPort > 0 { // only peer is cone should prepare t.ra
-}
+		var err error
-
+		t.ra, err = net.ResolveUDPAddr("udp", fmt.Sprintf("%s:%d", t.config.peerIP, t.config.peerConeNatPort))
-func (t *P2PTunnel) start() error {
+		if err != nil {
-	if t.config.linkMode == LinkModeUDPPunch {
+			return err
-		if err := t.handshake(); err != nil {
+		}
-			return err
+	}
-		}
+	if compareVersion(t.config.peerVersion, SyncServerTimeVersion) < 0 {
-	}
+		gLog.Printf(LvDEBUG, "peer version %s less than %s", t.config.peerVersion, SyncServerTimeVersion)
-	err := t.connectUnderlay()
+	} else {
-	if err != nil {
+		ts := time.Duration(int64(t.punchTs) + t.pn.dt + t.pn.ddtma*int64(time.Since(t.pn.hbTime)+PunchTsDelay)/int64(NetworkHeartbeatTime) - time.Now().UnixNano())
-		gLog.Println(LvERROR, err)
+		if ts > PunchTsDelay || ts < 0 {
-		return err
+			ts = PunchTsDelay
-	}
+		}
-	return nil
+		gLog.Printf(LvDEBUG, "sleep %d ms", ts/time.Millisecond)
-}
+		time.Sleep(ts)
-
+	}
-func (t *P2PTunnel) handshake() error {
+	gLog.Println(LvDEBUG, "handshake to ", t.config.LogPeerNode())
-	if t.config.peerConeNatPort > 0 { // only peer is cone should prepare t.ra
+	var err error
-		var err error
+	if gConf.Network.natType == NATCone && t.config.peerNatType == NATCone {
-		t.ra, err = net.ResolveUDPAddr("udp", fmt.Sprintf("%s:%d", t.config.peerIP, t.config.peerConeNatPort))
+		err = handshakeC2C(t)
-		if err != nil {
+	} else if t.config.peerNatType == NATSymmetric && gConf.Network.natType == NATSymmetric {
-			return err
+		err = ErrorS2S
-		}
+		t.close()
-	}
+	} else if t.config.peerNatType == NATSymmetric && gConf.Network.natType == NATCone {
-	gLog.Println(LvDEBUG, "handshake to ", t.config.PeerNode)
+		err = handshakeC2S(t)
-	var err error
+	} else if t.config.peerNatType == NATCone && gConf.Network.natType == NATSymmetric {
-	// TODO: handle NATNone, nodes with public ip has no punching
+		err = handshakeS2C(t)
-	if t.pn.config.natType == NATCone && t.config.peerNatType == NATCone {
+	} else {
-		err = handshakeC2C(t)
+		return errors.New("unknown error")
-	} else if t.config.peerNatType == NATSymmetric && t.pn.config.natType == NATSymmetric {
+	}
-		err = ErrorS2S
+	if err != nil {
-		t.close()
+		gLog.Println(LvERROR, "punch handshake error:", err)
-	} else if t.config.peerNatType == NATSymmetric && t.pn.config.natType == NATCone {
+		return err
-		err = handshakeC2S(t)
+	}
-	} else if t.config.peerNatType == NATCone && t.pn.config.natType == NATSymmetric {
+	gLog.Printf(LvDEBUG, "handshake to %s ok", t.config.LogPeerNode())
-		err = handshakeS2C(t)
+	return nil
-	} else {
+}
-		return errors.New("unknown error")
+
-	}
+func (t *P2PTunnel) connectUnderlay() (err error) {
-	if err != nil {
+	switch t.config.linkMode {
-		gLog.Println(LvERROR, "punch handshake error:", err)
+	case LinkModeTCP6:
-		return err
+		t.conn, err = t.connectUnderlayTCP6()
-	}
+	case LinkModeTCP4:
-	gLog.Printf(LvDEBUG, "handshake to %s ok", t.config.PeerNode)
+		t.conn, err = t.connectUnderlayTCP()
-	return nil
+	case LinkModeTCPPunch:
-}
+		if gConf.Network.natType == NATSymmetric || t.config.peerNatType == NATSymmetric {
-
+			t.conn, err = t.connectUnderlayTCPSymmetric()
-func (t *P2PTunnel) connectUnderlay() (err error) {
+		} else {
-	switch t.config.linkMode {
+			t.conn, err = t.connectUnderlayTCP()
-	case LinkModeTCP6:
+		}
-		t.conn, err = t.connectUnderlayTCP6()
+	case LinkModeIntranet:
-	case LinkModeTCP4:
+		t.conn, err = t.connectUnderlayTCP()
-		t.conn, err = t.connectUnderlayTCP()
+	case LinkModeUDPPunch:
-	case LinkModeTCPPunch:
+		t.conn, err = t.connectUnderlayUDP()
-		t.conn, err = t.connectUnderlayTCP()
+
-	case LinkModeUDPPunch:
+	}
-		t.conn, err = t.connectUnderlayQuic()
+	if err != nil {
-
+		return err
-	}
+	}
-	if err != nil {
+	if t.conn == nil {
-		return err
+		return errors.New("connect underlay error")
-	}
+	}
-	if t.conn == nil {
+	t.setRun(true)
-		return errors.New("connect underlay error")
+	go t.readLoop()
-	}
+	go t.writeLoop()
-	t.setRun(true)
+	return nil
-	go t.readLoop()
+}
-	go t.heartbeatLoop()
+
-	return nil
+func (t *P2PTunnel) connectUnderlayUDP() (c underlay, err error) {
-}
+	gLog.Printf(LvDEBUG, "connectUnderlayUDP %s start ", t.config.LogPeerNode())
-
+	defer gLog.Printf(LvDEBUG, "connectUnderlayUDP %s end ", t.config.LogPeerNode())
-func (t *P2PTunnel) connectUnderlayQuic() (c underlay, err error) {
+	var ul underlay
-	gLog.Println(LvINFO, "connectUnderlayQuic start")
+	underlayProtocol := t.config.UnderlayProtocol
-	defer gLog.Println(LvINFO, "connectUnderlayQuic end")
+	if underlayProtocol == "" {
-	var qConn *underlayQUIC
+		underlayProtocol = "quic"
-	if t.config.isUnderlayServer == 1 {
+	}
-		time.Sleep(time.Millisecond * 10) // punching udp port will need some times in some env
+	if t.config.isUnderlayServer == 1 {
-		qConn, err = listenQuic(t.la.String(), TunnelIdleTimeout)
+		time.Sleep(time.Millisecond * 10) // punching udp port will need some times in some env
-		if err != nil {
+		go t.pn.push(t.config.PeerNode, MsgPushUnderlayConnect, nil)
-			gLog.Println(LvINFO, "listen quic error:", err, ", retry...")
+		if t.config.UnderlayProtocol == "kcp" {
-		}
+			ul, err = listenKCP(t.la.String(), TunnelIdleTimeout)
-		t.pn.push(t.config.PeerNode, MsgPushUnderlayConnect, nil)
+		} else {
-		err = qConn.Accept()
+			ul, err = listenQuic(t.la.String(), TunnelIdleTimeout)
-		if err != nil {
+		}
-			qConn.CloseListener()
+
-			return nil, fmt.Errorf("accept quic error:%s", err)
+		if err != nil {
-		}
+			gLog.Printf(LvINFO, "listen %s error:%s", underlayProtocol, err)
-		_, buff, err := qConn.ReadBuffer()
+			return nil, err
-		if err != nil {
+		}
-			qConn.listener.Close()
+
-			return nil, fmt.Errorf("read start msg error:%s", err)
+		_, buff, err := ul.ReadBuffer()
-		}
+		if err != nil {
-		if buff != nil {
+			ul.Close()
-			gLog.Println(LvDEBUG, string(buff))
+			return nil, fmt.Errorf("read start msg error:%s", err)
-		}
+		}
-		qConn.WriteBytes(MsgP2P, MsgTunnelHandshakeAck, []byte("OpenP2P,hello2"))
+		if buff != nil {
-		gLog.Println(LvDEBUG, "quic connection ok")
+			gLog.Println(LvDEBUG, string(buff))
-		return qConn, nil
+		}
-	}
+		ul.WriteBytes(MsgP2P, MsgTunnelHandshakeAck, []byte("OpenP2P,hello2"))
-
+		gLog.Printf(LvDEBUG, "%s connection ok", underlayProtocol)
-	//else
+		return ul, nil
-	conn, e := net.ListenUDP("udp", t.la)
+	}
-	if e != nil {
+
-		time.Sleep(time.Millisecond * 10)
+	//else
-		conn, e = net.ListenUDP("udp", t.la)
+	conn, errL := net.ListenUDP("udp", t.la)
-		if e != nil {
+	if errL != nil {
-			return nil, fmt.Errorf("quic listen error:%s", e)
+		time.Sleep(time.Millisecond * 10)
-		}
+		conn, errL = net.ListenUDP("udp", t.la)
-	}
+		if errL != nil {
-	t.pn.read(t.config.PeerNode, MsgPush, MsgPushUnderlayConnect, time.Second*5)
+			return nil, fmt.Errorf("%s listen error:%s", underlayProtocol, errL)
-	gLog.Println(LvDEBUG, "quic dial to ", t.ra.String())
+		}
-	qConn, e = dialQuic(conn, t.ra, TunnelIdleTimeout)
+	}
-	if e != nil {
+	t.pn.read(t.config.PeerNode, MsgPush, MsgPushUnderlayConnect, ReadMsgTimeout)
-		return nil, fmt.Errorf("quic dial to %s error:%s", t.ra.String(), e)
+	gLog.Printf(LvDEBUG, "%s dial to %s", underlayProtocol, t.ra.String())
-	}
+	if t.config.UnderlayProtocol == "kcp" {
-	handshakeBegin := time.Now()
+		ul, errL = dialKCP(conn, t.ra, TunnelIdleTimeout)
-	qConn.WriteBytes(MsgP2P, MsgTunnelHandshake, []byte("OpenP2P,hello"))
+	} else {
-	_, buff, err := qConn.ReadBuffer()
+		ul, errL = dialQuic(conn, t.ra, TunnelIdleTimeout)
-	if e != nil {
+	}
-		qConn.listener.Close()
+
-		return nil, fmt.Errorf("read MsgTunnelHandshake error:%s", err)
+	if errL != nil {
-	}
+		return nil, fmt.Errorf("%s dial to %s error:%s", underlayProtocol, t.ra.String(), errL)
-	if buff != nil {
+	}
-		gLog.Println(LvDEBUG, string(buff))
+	handshakeBegin := time.Now()
-	}
+	ul.WriteBytes(MsgP2P, MsgTunnelHandshake, []byte("OpenP2P,hello"))
-
+	_, buff, err := ul.ReadBuffer() // TODO: kcp need timeout
-	gLog.Println(LvINFO, "rtt=", time.Since(handshakeBegin))
+	if err != nil {
-	gLog.Println(LvDEBUG, "quic connection ok")
+		ul.Close()
-	t.linkModeWeb = LinkModeUDPPunch
+		return nil, fmt.Errorf("read MsgTunnelHandshake error:%s", err)
-	return qConn, nil
+	}
-}
+	if buff != nil {
-
+		gLog.Println(LvDEBUG, string(buff))
-// websocket
+	}
-func (t *P2PTunnel) connectUnderlayTCP() (c underlay, err error) {
+
-	gLog.Println(LvINFO, "connectUnderlayTCP start")
+	gLog.Println(LvINFO, "rtt=", time.Since(handshakeBegin))
-	defer gLog.Println(LvINFO, "connectUnderlayTCP end")
+	gLog.Printf(LvINFO, "%s connection ok", underlayProtocol)
-	var qConn *underlayTCP
+	t.linkModeWeb = LinkModeUDPPunch
-	if t.config.isUnderlayServer == 1 {
+	return ul, nil
-		t.pn.push(t.config.PeerNode, MsgPushUnderlayConnect, nil)
+}
-		qConn, err = listenTCP(t.config.peerIP, t.config.peerConeNatPort, t.coneLocalPort, t.config.linkMode)
+
-		if err != nil {
+func (t *P2PTunnel) connectUnderlayTCP() (c underlay, err error) {
-			return nil, fmt.Errorf("listen TCP error:%s", err)
+	gLog.Printf(LvDEBUG, "connectUnderlayTCP %s start ", t.config.LogPeerNode())
-		}
+	defer gLog.Printf(LvDEBUG, "connectUnderlayTCP %s end ", t.config.LogPeerNode())
-
+	var ul *underlayTCP
-		_, buff, err := qConn.ReadBuffer()
+	peerIP := t.config.peerIP
-		if err != nil {
+	if t.config.linkMode == LinkModeIntranet {
-			return nil, fmt.Errorf("read start msg error:%s", err)
+		peerIP = t.config.peerLanIP
-		}
+	}
-		if buff != nil {
+	// server side
-			gLog.Println(LvDEBUG, string(buff))
+	if t.config.isUnderlayServer == 1 {
-		}
+		ul, err = listenTCP(peerIP, t.config.peerConeNatPort, t.coneLocalPort, t.config.linkMode, t)
-		qConn.WriteBytes(MsgP2P, MsgTunnelHandshakeAck, []byte("OpenP2P,hello2"))
+		if err != nil {
-		gLog.Println(LvINFO, "TCP connection ok")
+			return nil, fmt.Errorf("listen TCP error:%s", err)
-		return qConn, nil
+		}
-	}
+		gLog.Println(LvINFO, "TCP connection ok")
-
+		t.linkModeWeb = LinkModeIPv4
-	//else
+		if t.config.linkMode == LinkModeIntranet {
-	t.pn.read(t.config.PeerNode, MsgPush, MsgPushUnderlayConnect, time.Second*5)
+			t.linkModeWeb = LinkModeIntranet
-	gLog.Println(LvDEBUG, "TCP dial to ", t.config.peerIP, ":", t.config.peerConeNatPort)
+		}
-	qConn, err = dialTCP(t.config.peerIP, t.config.peerConeNatPort, t.coneLocalPort, t.config.linkMode)
+		return ul, nil
-	if err != nil {
+	}
-		return nil, fmt.Errorf("TCP dial to %s:%d error:%s", t.config.peerIP, t.config.peerConeNatPort, err)
+
-	}
+	// client side
-	handshakeBegin := time.Now()
+	if t.config.linkMode == LinkModeTCP4 {
-	qConn.WriteBytes(MsgP2P, MsgTunnelHandshake, []byte("OpenP2P,hello"))
+		t.pn.read(t.config.PeerNode, MsgPush, MsgPushUnderlayConnect, ReadMsgTimeout)
-	_, buff, err := qConn.ReadBuffer()
+	} else { //tcp punch should sleep for punch the same time
-	if err != nil {
+		if compareVersion(t.config.peerVersion, SyncServerTimeVersion) < 0 {
-		return nil, fmt.Errorf("read MsgTunnelHandshake error:%s", err)
+			gLog.Printf(LvDEBUG, "peer version %s less than %s", t.config.peerVersion, SyncServerTimeVersion)
-	}
+		} else {
-	if buff != nil {
+			ts := time.Duration(int64(t.punchTs) + t.pn.dt + t.pn.ddtma*int64(time.Since(t.pn.hbTime)+PunchTsDelay)/int64(NetworkHeartbeatTime) - time.Now().UnixNano())
-		gLog.Println(LvDEBUG, string(buff))
+			if ts > PunchTsDelay || ts < 0 {
-	}
+				ts = PunchTsDelay
-
+			}
-	gLog.Println(LvINFO, "rtt=", time.Since(handshakeBegin))
+			gLog.Printf(LvDEBUG, "sleep %d ms", ts/time.Millisecond)
-	gLog.Println(LvINFO, "TCP connection ok")
+			time.Sleep(ts)
-	t.linkModeWeb = LinkModeIPv4
+		}
-	return qConn, nil
+	}
-}
+	ul, err = dialTCP(peerIP, t.config.peerConeNatPort, t.coneLocalPort, t.config.linkMode)
-
+	if err != nil {
-func (t *P2PTunnel) connectUnderlayTCP6() (c underlay, err error) {
+		return nil, fmt.Errorf("TCP dial to %s:%d error:%s", t.config.peerIP, t.config.peerConeNatPort, err)
-	gLog.Println(LvINFO, "connectUnderlayTCP6 start")
+	}
-	defer gLog.Println(LvINFO, "connectUnderlayTCP6 end")
+	handshakeBegin := time.Now()
-	var qConn *underlayTCP6
+	tidBuff := new(bytes.Buffer)
-	if t.config.isUnderlayServer == 1 {
+	binary.Write(tidBuff, binary.LittleEndian, t.id)
-		t.pn.push(t.config.PeerNode, MsgPushUnderlayConnect, nil)
+	ul.WriteBytes(MsgP2P, MsgTunnelHandshake, tidBuff.Bytes()) //  tunnelID
-		qConn, err = listenTCP6(t.coneNatPort, TunnelIdleTimeout)
+	_, buff, err := ul.ReadBuffer()
-		if err != nil {
+	if err != nil {
-			return nil, fmt.Errorf("listen TCP6 error:%s", err)
+		return nil, fmt.Errorf("read MsgTunnelHandshake error:%s", err)
-		}
+	}
-		_, buff, err := qConn.ReadBuffer()
+	if buff != nil {
-		if err != nil {
+		gLog.Println(LvDEBUG, "hello ", string(buff))
-			qConn.listener.Close()
+	}
-			return nil, fmt.Errorf("read start msg error:%s", err)
+
-		}
+	gLog.Println(LvINFO, "rtt=", time.Since(handshakeBegin))
-		if buff != nil {
+	gLog.Println(LvINFO, "TCP connection ok")
-			gLog.Println(LvDEBUG, string(buff))
+	t.linkModeWeb = LinkModeIPv4
-		}
+	if t.config.linkMode == LinkModeIntranet {
-		qConn.WriteBytes(MsgP2P, MsgTunnelHandshakeAck, []byte("OpenP2P,hello2"))
+		t.linkModeWeb = LinkModeIntranet
-		gLog.Println(LvDEBUG, "TCP6 connection ok")
+	}
-		return qConn, nil
+	return ul, nil
-	}
+}
-
+
-	//else
+func (t *P2PTunnel) connectUnderlayTCPSymmetric() (c underlay, err error) {
-	t.pn.read(t.config.PeerNode, MsgPush, MsgPushUnderlayConnect, time.Second*5)
+	gLog.Printf(LvDEBUG, "connectUnderlayTCPSymmetric %s start ", t.config.LogPeerNode())
-	gLog.Println(LvDEBUG, "TCP6 dial to ", t.config.peerIPv6)
+	defer gLog.Printf(LvDEBUG, "connectUnderlayTCPSymmetric %s end ", t.config.LogPeerNode())
-	qConn, err = dialTCP6(t.config.peerIPv6, t.config.peerConeNatPort)
+	ts := time.Duration(int64(t.punchTs) + t.pn.dt + t.pn.ddtma*int64(time.Since(t.pn.hbTime)+PunchTsDelay)/int64(NetworkHeartbeatTime) - time.Now().UnixNano())
-	if err != nil {
+	if ts > PunchTsDelay || ts < 0 {
-		return nil, fmt.Errorf("TCP6 dial to %s:%d error:%s", t.config.peerIPv6, t.config.peerConeNatPort, err)
+		ts = PunchTsDelay
-	}
+	}
-	handshakeBegin := time.Now()
+	gLog.Printf(LvDEBUG, "sleep %d ms", ts/time.Millisecond)
-	qConn.WriteBytes(MsgP2P, MsgTunnelHandshake, []byte("OpenP2P,hello"))
+	time.Sleep(ts)
-	_, buff, err := qConn.ReadBuffer()
+	startTime := time.Now()
-	if err != nil {
+	t.linkModeWeb = LinkModeTCPPunch
-		qConn.listener.Close()
+	gotCh := make(chan *underlayTCP, 1)
-		return nil, fmt.Errorf("read MsgTunnelHandshake error:%s", err)
+	var wg sync.WaitGroup
-	}
+	var success atomic.Int32
-	if buff != nil {
+	if t.config.peerNatType == NATSymmetric { // c2s
-		gLog.Println(LvDEBUG, string(buff))
+		randPorts := rand.Perm(65532)
-	}
+		for i := 0; i < SymmetricHandshakeNum; i++ {
-
+			wg.Add(1)
-	gLog.Println(LvINFO, "rtt=", time.Since(handshakeBegin))
+			go func(port int) {
-	gLog.Println(LvDEBUG, "TCP6 connection ok")
+				defer wg.Done()
-	t.linkModeWeb = LinkModeIPv6
+				ul, err := dialTCP(t.config.peerIP, port, t.coneLocalPort, LinkModeTCPPunch)
-	return qConn, nil
+				if err != nil {
-}
+					return
-
+				}
-func (t *P2PTunnel) readLoop() {
+				if !success.CompareAndSwap(0, 1) {
-	decryptData := make([]byte, ReadBuffLen+PaddingSize) // 16 bytes for padding
+					ul.Close() // only cone side close
-	gLog.Printf(LvDEBUG, "%d tunnel readloop start", t.id)
+					return
-	for t.isRuning() {
+				}
-		t.conn.SetReadDeadline(time.Now().Add(TunnelIdleTimeout))
+				err = ul.WriteMessage(MsgP2P, MsgPunchHandshakeAck, P2PHandshakeReq{ID: t.id})
-		head, body, err := t.conn.ReadBuffer()
+				if err != nil {
-		if err != nil {
+					ul.Close()
-			if t.isRuning() {
+					return
-				gLog.Printf(LvERROR, "%d tunnel read error:%s", t.id, err)
+				}
-			}
+				_, buff, err := ul.ReadBuffer()
-			break
+				if err != nil {
-		}
+					gLog.Printf(LvERROR, "utcp.ReadBuffer error:", err)
-		if head.MainType != MsgP2P {
+					return
-			continue
+				}
-		}
+				req := P2PHandshakeReq{}
-		switch head.SubType {
+				if err = json.Unmarshal(buff, &req); err != nil {
-		case MsgTunnelHeartbeat:
+					return
-			t.conn.WriteBytes(MsgP2P, MsgTunnelHeartbeatAck, nil)
+				}
-			gLog.Printf(LvDEBUG, "%d read tunnel heartbeat", t.id)
+				if req.ID != t.id {
-		case MsgTunnelHeartbeatAck:
+					return
-			t.hbMtx.Lock()
+				}
-			t.hbTime = time.Now()
+				gLog.Printf(LvINFO, "handshakeS2C TCP ok. cost %dms", time.Since(startTime)/time.Millisecond)
-			t.hbMtx.Unlock()
+
-			gLog.Printf(LvDEBUG, "%d read tunnel heartbeat ack", t.id)
+				gotCh <- ul
-		case MsgOverlayData:
+				close(gotCh)
-			if len(body) < overlayHeaderSize {
+			}(randPorts[i] + 2)
-				continue
+		}
-			}
+
-			overlayID := binary.LittleEndian.Uint64(body[:8])
+	} else { // s2c
-			gLog.Printf(LvDEBUG, "%d tunnel read overlay data %d", t.id, overlayID)
+		for i := 0; i < SymmetricHandshakeNum; i++ {
-			s, ok := t.overlayConns.Load(overlayID)
+			wg.Add(1)
-			if !ok {
+			go func() {
-				// debug level, when overlay connection closed, always has some packet not found tunnel
+				defer wg.Done()
-				gLog.Printf(LvDEBUG, "%d tunnel not found overlay connection %d", t.id, overlayID)
+				ul, err := dialTCP(t.config.peerIP, t.config.peerConeNatPort, 0, LinkModeTCPPunch)
-				continue
+				if err != nil {
-			}
+					return
-			overlayConn, ok := s.(*overlayConn)
+				}
-			if !ok {
+
-				continue
+				_, buff, err := ul.ReadBuffer()
-			}
+				if err != nil {
-			payload := body[overlayHeaderSize:]
+					gLog.Printf(LvERROR, "utcp.ReadBuffer error:", err)
-			var err error
+					return
-			if overlayConn.appKey != 0 {
+				}
-				payload, _ = decryptBytes(overlayConn.appKeyBytes, decryptData, body[overlayHeaderSize:], int(head.DataLen-uint32(overlayHeaderSize)))
+				req := P2PHandshakeReq{}
-			}
+				if err = json.Unmarshal(buff, &req); err != nil {
-			_, err = overlayConn.Write(payload)
+					return
-			if err != nil {
+				}
-				gLog.Println(LvERROR, "overlay write error:", err)
+				if req.ID != t.id {
-			}
+					return
-		case MsgRelayData:
+				}
-			gLog.Printf(LvDEBUG, "got relay data datalen=%d", head.DataLen)
+				err = ul.WriteMessage(MsgP2P, MsgPunchHandshakeAck, P2PHandshakeReq{ID: t.id})
-			if len(body) < 8 {
+				if err != nil {
-				continue
+					ul.Close()
-			}
+					return
-			tunnelID := binary.LittleEndian.Uint64(body[:8])
+				}
-			t.pn.relay(tunnelID, body[8:])
+				if success.CompareAndSwap(0, 1) {
-		case MsgRelayHeartbeat:
+					gotCh <- ul
-			req := RelayHeartbeat{}
+					close(gotCh)
-			err := json.Unmarshal(body, &req)
+				}
-			if err != nil {
+			}()
-				gLog.Printf(LvERROR, "wrong RelayHeartbeat:%s", err)
+		}
-				continue
+	}
-			}
+	select {
-			gLog.Printf(LvDEBUG, "got MsgRelayHeartbeat from %d:%d", req.RelayTunnelID, req.AppID)
+	case <-time.After(HandshakeTimeout):
-			relayHead := new(bytes.Buffer)
+		return nil, fmt.Errorf("wait tcp handshake timeout")
-			binary.Write(relayHead, binary.LittleEndian, req.RelayTunnelID)
+	case ul := <-gotCh:
-			msg, _ := newMessage(MsgP2P, MsgRelayHeartbeatAck, &req)
+		return ul, nil
-			msgWithHead := append(relayHead.Bytes(), msg...)
+	}
-			t.conn.WriteBytes(MsgP2P, MsgRelayData, msgWithHead)
+}
-		case MsgRelayHeartbeatAck:
+
-			req := RelayHeartbeat{}
+func (t *P2PTunnel) connectUnderlayTCP6() (c underlay, err error) {
-			err := json.Unmarshal(body, &req)
+	gLog.Printf(LvDEBUG, "connectUnderlayTCP6 %s start ", t.config.LogPeerNode())
-			if err != nil {
+	defer gLog.Printf(LvDEBUG, "connectUnderlayTCP6 %s end ", t.config.LogPeerNode())
-				gLog.Printf(LvERROR, "wrong RelayHeartbeat:%s", err)
+	var ul *underlayTCP6
-				continue
+	if t.config.isUnderlayServer == 1 {
-			}
+		t.pn.push(t.config.PeerNode, MsgPushUnderlayConnect, nil)
-			gLog.Printf(LvDEBUG, "got MsgRelayHeartbeatAck to %d", req.AppID)
+		ul, err = listenTCP6(t.coneNatPort, UnderlayConnectTimeout)
-			t.pn.updateAppHeartbeat(req.AppID)
+		if err != nil {
-		case MsgOverlayConnectReq:
+			return nil, fmt.Errorf("listen TCP6 error:%s", err)
-			req := OverlayConnectReq{}
+		}
-			err := json.Unmarshal(body, &req)
+		_, buff, err := ul.ReadBuffer()
-			if err != nil {
+		if err != nil {
-				gLog.Printf(LvERROR, "wrong MsgOverlayConnectReq:%s", err)
+			return nil, fmt.Errorf("read start msg error:%s", err)
-				continue
+		}
-			}
+		if buff != nil {
-			// app connect only accept token(not relay totp token), avoid someone using the share relay node's token
+			gLog.Println(LvDEBUG, string(buff))
-			if req.Token != t.pn.config.Token {
+		}
-				gLog.Println(LvERROR, "Access Denied:", req.Token)
+		ul.WriteBytes(MsgP2P, MsgTunnelHandshakeAck, []byte("OpenP2P,hello2"))
-				continue
+		gLog.Println(LvDEBUG, "TCP6 connection ok")
-			}
+		t.linkModeWeb = LinkModeIPv6
-
+		return ul, nil
-			overlayID := req.ID
+	}
-			gLog.Printf(LvDEBUG, "App:%d overlayID:%d connect %+v", req.AppID, overlayID, req)
+
-			oConn := overlayConn{
+	//else
-				tunnel:   t,
+	t.pn.read(t.config.PeerNode, MsgPush, MsgPushUnderlayConnect, ReadMsgTimeout)
-				id:       overlayID,
+	gLog.Println(LvDEBUG, "TCP6 dial to ", t.config.peerIPv6)
-				isClient: false,
+	ul, err = dialTCP6(t.config.peerIPv6, t.config.peerConeNatPort)
-				rtid:     req.RelayTunnelID,
+	if err != nil {
-				appID:    req.AppID,
+		return nil, fmt.Errorf("TCP6 dial to %s:%d error:%s", t.config.peerIPv6, t.config.peerConeNatPort, err)
-				appKey:   GetKey(req.AppID),
+	}
-			}
+	handshakeBegin := time.Now()
-			if req.Protocol == "udp" {
+	ul.WriteBytes(MsgP2P, MsgTunnelHandshake, []byte("OpenP2P,hello"))
-				oConn.connUDP, err = net.DialUDP("udp", nil, &net.UDPAddr{IP: net.ParseIP(req.DstIP), Port: req.DstPort})
+	_, buff, err := ul.ReadBuffer()
-			} else {
+	if err != nil {
-				oConn.connTCP, err = net.DialTimeout("tcp", fmt.Sprintf("%s:%d", req.DstIP, req.DstPort), time.Second*5)
+		return nil, fmt.Errorf("read MsgTunnelHandshake error:%s", err)
-			}
+	}
-			if err != nil {
+	if buff != nil {
-				gLog.Println(LvERROR, err)
+		gLog.Println(LvDEBUG, string(buff))
-				continue
+	}
-			}
+
-
+	gLog.Println(LvINFO, "rtt=", time.Since(handshakeBegin))
-			// calc key bytes for encrypt
+	gLog.Println(LvINFO, "TCP6 connection ok")
-			if oConn.appKey != 0 {
+	t.linkModeWeb = LinkModeIPv6
-				encryptKey := make([]byte, 16)
+	return ul, nil
-				binary.LittleEndian.PutUint64(encryptKey, oConn.appKey)
+}
-				binary.LittleEndian.PutUint64(encryptKey[8:], oConn.appKey)
+
-				oConn.appKeyBytes = encryptKey
+func (t *P2PTunnel) readLoop() {
-			}
+	decryptData := make([]byte, ReadBuffLen+PaddingSize) // 16 bytes for padding
-
+	gLog.Printf(LvDEBUG, "%d tunnel readloop start", t.id)
-			t.overlayConns.Store(oConn.id, &oConn)
+	for t.isRuning() {
-			go oConn.run()
+		t.conn.SetReadDeadline(time.Now().Add(TunnelHeartbeatTime * 2))
-		case MsgOverlayDisconnectReq:
+		head, body, err := t.conn.ReadBuffer()
-			req := OverlayDisconnectReq{}
+		if err != nil {
-			err := json.Unmarshal(body, &req)
+			if t.isRuning() {
-			if err != nil {
+				gLog.Printf(LvERROR, "%d tunnel read error:%s", t.id, err)
-				gLog.Printf(LvERROR, "wrong OverlayDisconnectRequest:%s", err)
+			}
-				continue
+			break
-			}
+		}
-			overlayID := req.ID
+		if head.MainType != MsgP2P {
-			gLog.Printf(LvDEBUG, "%d disconnect overlay connection %d", t.id, overlayID)
+			gLog.Printf(LvWARN, "%d head.MainType != MsgP2P", t.id)
-			i, ok := t.overlayConns.Load(overlayID)
+			continue
-			if ok {
+		}
-				oConn := i.(*overlayConn)
+		// TODO: replace some case implement to functions
-				oConn.running = false
+		switch head.SubType {
-			}
+		case MsgTunnelHeartbeat:
-		default:
+			t.hbMtx.Lock()
-		}
+			t.hbTime = time.Now()
-	}
+			t.hbMtx.Unlock()
-	t.setRun(false)
+			t.conn.WriteBytes(MsgP2P, MsgTunnelHeartbeatAck, nil)
-	t.conn.Close()
+			gLog.Printf(LvDev, "%d read tunnel heartbeat", t.id)
-	gLog.Printf(LvDEBUG, "%d tunnel readloop end", t.id)
+		case MsgTunnelHeartbeatAck:
-}
+			t.hbMtx.Lock()
-
+			t.hbTime = time.Now()
-func (t *P2PTunnel) heartbeatLoop() {
+			t.hbMtx.Unlock()
-	tc := time.NewTicker(TunnelHeartbeatTime)
+			gLog.Printf(LvDev, "%d read tunnel heartbeat ack", t.id)
-	defer tc.Stop()
+		case MsgOverlayData:
-	gLog.Printf(LvDEBUG, "%d tunnel heartbeatLoop start", t.id)
+			if len(body) < overlayHeaderSize {
-	defer gLog.Printf(LvDEBUG, "%d tunnel heartbeatLoop end", t.id)
+				gLog.Printf(LvWARN, "%d len(body) < overlayHeaderSize", t.id)
-	for t.isRuning() {
+				continue
-		select {
+			}
-		case <-tc.C:
+			overlayID := binary.LittleEndian.Uint64(body[:8])
-			// tunnel send
+			gLog.Printf(LvDev, "%d tunnel read overlay data %d bodylen=%d", t.id, overlayID, head.DataLen)
-			err := t.conn.WriteBytes(MsgP2P, MsgTunnelHeartbeat, nil)
+			s, ok := t.overlayConns.Load(overlayID)
-			if err != nil {
+			if !ok {
-				gLog.Printf(LvERROR, "%d write tunnel heartbeat error %s", t.id, err)
+				// debug level, when overlay connection closed, always has some packet not found tunnel
-				t.setRun(false)
+				gLog.Printf(LvDEBUG, "%d tunnel not found overlay connection %d", t.id, overlayID)
-				return
+				continue
-			}
+			}
-			gLog.Printf(LvDEBUG, "%d write tunnel heartbeat ok", t.id)
+			overlayConn, ok := s.(*overlayConn)
-		}
+			if !ok {
-	}
+				continue
-}
+			}
-
+			payload := body[overlayHeaderSize:]
-func (t *P2PTunnel) listen() error {
+			var err error
-	// notify client to connect
+			if overlayConn.appKey != 0 {
-	rsp := PushConnectRsp{
+				payload, _ = decryptBytes(overlayConn.appKeyBytes, decryptData, body[overlayHeaderSize:], int(head.DataLen-uint32(overlayHeaderSize)))
-		Error:   0,
+			}
-		Detail:  "connect ok",
+			_, err = overlayConn.Write(payload)
-		To:      t.config.PeerNode,
+			if err != nil {
-		From:    t.pn.config.Node,
+				gLog.Println(LvERROR, "overlay write error:", err)
-		NatType: t.pn.config.natType,
+			}
-		HasIPv4: t.pn.config.hasIPv4,
+		case MsgNodeData:
-		// IPv6:            t.pn.config.IPv6,
+			t.handleNodeData(head, body, false)
-		HasUPNPorNATPMP: t.pn.config.hasUPNPorNATPMP,
+		case MsgRelayNodeData:
-		FromIP:          t.pn.config.publicIP,
+			t.handleNodeData(head, body, true)
-		ConeNatPort:     t.coneNatPort,
+		case MsgRelayData:
-		ID:              t.id,
+			if len(body) < 8 {
-		Version:         OpenP2PVersion,
+				continue
-	}
+			}
-	// only private node set ipv6
+			tunnelID := binary.LittleEndian.Uint64(body[:8])
-	if t.config.fromToken == t.pn.config.Token {
+			gLog.Printf(LvDev, "relay data to %d, len=%d", tunnelID, head.DataLen-RelayHeaderSize)
-		t.pn.refreshIPv6(false)
+			if err := t.pn.relay(tunnelID, body[RelayHeaderSize:]); err != nil {
-		rsp.IPv6 = t.pn.config.publicIPv6
+				gLog.Printf(LvERROR, "%s:%d relay to %d len=%d error:%s", t.config.LogPeerNode(), t.id, tunnelID, len(body), ErrRelayTunnelNotFound)
-	}
+			}
-
+		case MsgRelayHeartbeat:
-	t.pn.push(t.config.PeerNode, MsgPushConnectRsp, rsp)
+			req := RelayHeartbeat{}
-	gLog.Printf(LvDEBUG, "p2ptunnel wait for connecting")
+			if err := json.Unmarshal(body, &req); err != nil {
-	t.tunnelServer = true
+				gLog.Printf(LvERROR, "wrong %v:%s", reflect.TypeOf(req), err)
-	return t.start()
+				continue
-}
+			}
-
+			// TODO: debug relay heartbeat
-func (t *P2PTunnel) closeOverlayConns(appID uint64) {
+			gLog.Printf(LvDEBUG, "read MsgRelayHeartbeat from rtid:%d,appid:%d", req.RelayTunnelID, req.AppID)
-	t.overlayConns.Range(func(_, i interface{}) bool {
+			// update app hbtime
-		oConn := i.(*overlayConn)
+			t.pn.updateAppHeartbeat(req.AppID)
-		if oConn.appID == appID {
+			req.From = gConf.Network.Node
-			if oConn.connTCP != nil {
+			t.WriteMessage(req.RelayTunnelID, MsgP2P, MsgRelayHeartbeatAck, &req)
-				oConn.connTCP.Close()
+		case MsgRelayHeartbeatAck:
-				oConn.connTCP = nil
+			req := RelayHeartbeat{}
-			}
+			err := json.Unmarshal(body, &req)
-			if oConn.connUDP != nil {
+			if err != nil {
-				oConn.connUDP.Close()
+				gLog.Printf(LvERROR, "wrong RelayHeartbeat:%s", err)
-				oConn.connUDP = nil
+				continue
-			}
+			}
-		}
+			// TODO: debug relay heartbeat
-		return true
+			gLog.Printf(LvDEBUG, "read MsgRelayHeartbeatAck to appid:%d", req.AppID)
-	})
+			t.pn.updateAppHeartbeat(req.AppID)
-}
+		case MsgOverlayConnectReq:
+			req := OverlayConnectReq{}
+			if err := json.Unmarshal(body, &req); err != nil {
+				gLog.Printf(LvERROR, "wrong %v:%s", reflect.TypeOf(req), err)
+				continue
+			}
+			// app connect only accept token(not relay totp token), avoid someone using the share relay node's token
+			if req.Token != gConf.Network.Token {
+				gLog.Println(LvERROR, "Access Denied:", req.Token)
+				continue
+			}
+
+			overlayID := req.ID
+			gLog.Printf(LvDEBUG, "App:%d overlayID:%d connect %s:%d", req.AppID, overlayID, req.DstIP, req.DstPort)
+			oConn := overlayConn{
+				tunnel:   t,
+				id:       overlayID,
+				isClient: false,
+				rtid:     req.RelayTunnelID,
+				appID:    req.AppID,
+				appKey:   GetKey(req.AppID),
+				running:  true,
+			}
+			if req.Protocol == "udp" {
+				oConn.connUDP, err = net.DialUDP("udp", nil, &net.UDPAddr{IP: net.ParseIP(req.DstIP), Port: req.DstPort})
+			} else {
+				oConn.connTCP, err = net.DialTimeout("tcp", fmt.Sprintf("%s:%d", req.DstIP, req.DstPort), ReadMsgTimeout)
+
+			}
+			if err != nil {
+				gLog.Println(LvERROR, err)
+				continue
+			}
+
+			// calc key bytes for encrypt
+			if oConn.appKey != 0 {
+				encryptKey := make([]byte, AESKeySize)
+				binary.LittleEndian.PutUint64(encryptKey, oConn.appKey)
+				binary.LittleEndian.PutUint64(encryptKey[8:], oConn.appKey)
+				oConn.appKeyBytes = encryptKey
+			}
+
+			t.overlayConns.Store(oConn.id, &oConn)
+			go oConn.run()
+		case MsgOverlayDisconnectReq:
+			req := OverlayDisconnectReq{}
+			if err := json.Unmarshal(body, &req); err != nil {
+				gLog.Printf(LvERROR, "wrong %v:%s", reflect.TypeOf(req), err)
+				continue
+			}
+			overlayID := req.ID
+			gLog.Printf(LvDEBUG, "%d disconnect overlay connection %d", t.id, overlayID)
+			i, ok := t.overlayConns.Load(overlayID)
+			if ok {
+				oConn := i.(*overlayConn)
+				oConn.Close()
+			}
+		default:
+		}
+	}
+	t.close()
+	gLog.Printf(LvDEBUG, "%d tunnel readloop end", t.id)
+}
+
+func (t *P2PTunnel) writeLoop() {
+	t.hbMtx.Lock()
+	t.hbTime = time.Now() // init
+	t.hbMtx.Unlock()
+	tc := time.NewTicker(TunnelHeartbeatTime)
+	defer tc.Stop()
+	gLog.Printf(LvDEBUG, "%s:%d tunnel writeLoop start", t.config.LogPeerNode(), t.id)
+	defer gLog.Printf(LvDEBUG, "%s:%d tunnel writeLoop end", t.config.LogPeerNode(), t.id)
+	for t.isRuning() {
+		select {
+		case buff := <-t.writeDataSmall:
+			t.conn.WriteBuffer(buff)
+			// gLog.Printf(LvDEBUG, "write icmp %d", time.Now().Unix())
+		default:
+			select {
+			case buff := <-t.writeDataSmall:
+				t.conn.WriteBuffer(buff)
+				// gLog.Printf(LvDEBUG, "write icmp %d", time.Now().Unix())
+			case buff := <-t.writeData:
+				t.conn.WriteBuffer(buff)
+			case <-tc.C:
+				// tunnel send
+				err := t.conn.WriteBytes(MsgP2P, MsgTunnelHeartbeat, nil)
+				if err != nil {
+					gLog.Printf(LvERROR, "%d write tunnel heartbeat error %s", t.id, err)
+					t.close()
+					return
+				}
+				gLog.Printf(LvDev, "%d write tunnel heartbeat ok", t.id)
+			}
+		}
+	}
+}
+
+func (t *P2PTunnel) listen() error {
+	// notify client to connect
+	rsp := PushConnectRsp{
+		Error:   0,
+		Detail:  "connect ok",
+		To:      t.config.PeerNode,
+		From:    gConf.Network.Node,
+		NatType: gConf.Network.natType,
+		HasIPv4: gConf.Network.hasIPv4,
+		// IPv6:            gConf.Network.IPv6,
+		HasUPNPorNATPMP: gConf.Network.hasUPNPorNATPMP,
+		FromIP:          gConf.Network.publicIP,
+		ConeNatPort:     t.coneNatPort,
+		ID:              t.id,
+		PunchTs:         uint64(time.Now().UnixNano() + int64(PunchTsDelay) - t.pn.dt),
+		Version:         OpenP2PVersion,
+	}
+	t.punchTs = rsp.PunchTs
+	// only private node set ipv6
+	if t.config.fromToken == gConf.Network.Token {
+		rsp.IPv6 = gConf.IPv6()
+	}
+
+	t.pn.push(t.config.PeerNode, MsgPushConnectRsp, rsp)
+	gLog.Printf(LvDEBUG, "p2ptunnel wait for connecting")
+	t.tunnelServer = true
+	return t.start()
+}
+
+func (t *P2PTunnel) closeOverlayConns(appID uint64) {
+	t.overlayConns.Range(func(_, i interface{}) bool {
+		oConn := i.(*overlayConn)
+		if oConn.appID == appID {
+			oConn.Close()
+		}
+		return true
+	})
+}
+
+func (t *P2PTunnel) handleNodeData(head *openP2PHeader, body []byte, isRelay bool) {
+	gLog.Printf(LvDev, "%d tunnel read node data bodylen=%d, relay=%t", t.id, head.DataLen, isRelay)
+	ch := t.pn.nodeData
+	// if body[9] == 1 { // TODO: deal relay
+	// 	ch = t.pn.nodeDataSmall
+	// 	gLog.Printf(LvDEBUG, "read icmp %d", time.Now().Unix())
+	// }
+	if isRelay {
+		fromPeerID := binary.LittleEndian.Uint64(body[:8])
+		ch <- &NodeData{fromPeerID, body[8:]} // TODO: cache peerNodeID; encrypt/decrypt
+	} else {
+		ch <- &NodeData{NodeNameToID(t.config.PeerNode), body} // TODO: cache peerNodeID; encrypt/decrypt
+	}
+}
+
+func (t *P2PTunnel) asyncWriteNodeData(mainType, subType uint16, data []byte) {
+	writeBytes := append(encodeHeader(mainType, subType, uint32(len(data))), data...)
+	// if len(data) < 192 {
+	if data[9] == 1 { // icmp
+		select {
+		case t.writeDataSmall <- writeBytes:
+			// gLog.Printf(LvWARN, "%s:%d t.writeDataSmall write %d", t.config.PeerNode, t.id, len(t.writeDataSmall))
+		default:
+			gLog.Printf(LvWARN, "%s:%d t.writeDataSmall is full, drop it", t.config.LogPeerNode(), t.id)
+		}
+	} else {
+		select {
+		case t.writeData <- writeBytes:
+		default:
+			gLog.Printf(LvWARN, "%s:%d t.writeData is full, drop it", t.config.LogPeerNode(), t.id)
+		}
+	}
+
+}
+
+func (t *P2PTunnel) WriteMessage(rtid uint64, mainType uint16, subType uint16, req interface{}) error {
+	if rtid == 0 {
+		return t.conn.WriteMessage(mainType, subType, &req)
+	}
+	relayHead := new(bytes.Buffer)
+	binary.Write(relayHead, binary.LittleEndian, rtid)
+	msg, _ := newMessage(mainType, subType, &req)
+	msgWithHead := append(relayHead.Bytes(), msg...)
+	return t.conn.WriteBytes(mainType, MsgRelayData, msgWithHead)
+
+}

core/p2ptunnel_test.go

@@ -0,0 +1,24 @@
+package openp2p
+
+import (
+	"fmt"
+	"testing"
+)
+
+func TestSelectPriority(t *testing.T) {
+	writeData := make(chan []byte, WriteDataChanSize)
+	writeDataSmall := make(chan []byte, WriteDataChanSize/30)
+	for i := 0; i < 100; i++ {
+		writeData <- []byte("data")
+		writeDataSmall <- []byte("small data")
+	}
+	for i := 0; i < 100; i++ {
+		select {
+		case buff := <-writeDataSmall:
+			fmt.Printf("got small data:%s\n", string(buff))
+		case buff := <-writeData:
+			fmt.Printf("got data:%s\n", string(buff))
+		}
+	}
+
+}

core/protocol.go

@@ -10,9 +10,14 @@ import (
 	"time"
 )
 
-const OpenP2PVersion = "3.5.6"
+const OpenP2PVersion = "3.19.0"
 const ProductName string = "openp2p"
 const LeastSupportVersion = "3.0.0"
+const SyncServerTimeVersion = "3.9.0"
+const SymmetricSimultaneouslySendVersion = "3.10.7"
+const PublicIPVersion = "3.11.2"
+const SupportIntranetVersion = "3.14.5"
+const SupportDualTunnelVersion = "3.15.5"
 
 const (
 	IfconfigPort1 = 27180
@@ -37,6 +42,8 @@ type PushHeader struct {
 
 var PushHeaderSize = binary.Size(PushHeader{})
 
+const RelayHeaderSize = 8
+
 type overlayHeader struct {
 	id uint64
 }
@@ -77,25 +84,31 @@ const (
 	MsgRelay     = 5
 	MsgReport    = 6
 	MsgQuery     = 7
+	MsgSDWAN     = 8
 )
 
 // TODO: seperate node push and web push.
 const (
-	MsgPushRsp               = 0
+	MsgPushRsp                  = 0
-	MsgPushConnectReq        = 1
+	MsgPushConnectReq           = 1
-	MsgPushConnectRsp        = 2
+	MsgPushConnectRsp           = 2
-	MsgPushHandshakeStart    = 3
+	MsgPushHandshakeStart       = 3
-	MsgPushAddRelayTunnelReq = 4
+	MsgPushAddRelayTunnelReq    = 4
-	MsgPushAddRelayTunnelRsp = 5
+	MsgPushAddRelayTunnelRsp    = 5
-	MsgPushUpdate            = 6
+	MsgPushUpdate               = 6
-	MsgPushReportApps        = 7
+	MsgPushReportApps           = 7
-	MsgPushUnderlayConnect   = 8
+	MsgPushUnderlayConnect      = 8
-	MsgPushEditApp           = 9
+	MsgPushEditApp              = 9
-	MsgPushSwitchApp         = 10
+	MsgPushSwitchApp            = 10
-	MsgPushRestart           = 11
+	MsgPushRestart              = 11
-	MsgPushEditNode          = 12
+	MsgPushEditNode             = 12
-	MsgPushAPPKey            = 13
+	MsgPushAPPKey               = 13
-	MsgPushReportLog         = 14
+	MsgPushReportLog            = 14
+	MsgPushDstNodeOnline        = 15
+	MsgPushReportGoroutine      = 16
+	MsgPushReportMemApps        = 17
+	MsgPushServerSideSaveMemApp = 18
+	MsgPushCheckRemoteService   = 19
 )
 
 // MsgP2P sub type message
@@ -113,6 +126,8 @@ const (
 	MsgRelayData
 	MsgRelayHeartbeat
 	MsgRelayHeartbeatAck
+	MsgNodeData
+	MsgRelayNodeData
 )
 
 // MsgRelay sub type message
@@ -128,27 +143,40 @@ const (
 	MsgReportConnect
 	MsgReportApps
 	MsgReportLog
+	MsgReportMemApps
+	MsgReportResponse
 )
 
 const (
-	ReadBuffLen           = 4096             // for UDP maybe not enough
+	ReadBuffLen               = 4096 // for UDP maybe not enough
-	NetworkHeartbeatTime  = time.Second * 30 // TODO: server no response hb, save flow
+	NetworkHeartbeatTime      = time.Second * 30
-	TunnelHeartbeatTime   = time.Second * 15
+	TunnelHeartbeatTime       = time.Second * 10 // some nat udp session expired time less than 15s. change to 10s
-	TunnelIdleTimeout     = time.Minute
+	UnderlayTCPKeepalive      = time.Second * 5
-	SymmetricHandshakeNum = 800 // 0.992379
+	UnderlayTCPConnectTimeout = time.Second * 5
+	TunnelIdleTimeout         = time.Minute
+	SymmetricHandshakeNum     = 800 // 0.992379
 	// SymmetricHandshakeNum        = 1000 // 0.999510
-	SymmetricHandshakeInterval   = time.Millisecond
+	SymmetricHandshakeInterval = time.Millisecond
-	SymmetricHandshakeAckTimeout = time.Second * 11
+	HandshakeTimeout           = time.Second * 7
-	PeerAddRelayTimeount         = time.Second * 20
+	PunchTsDelay               = time.Second * 3
-	CheckActiveTimeout           = time.Second * 5
+	PeerAddRelayTimeount       = time.Second * 30 // peer need times. S2C\TCP\TCP Punch\UDP Punch
-	PaddingSize                  = 16
+	CheckActiveTimeout         = time.Second * 5
-	AESKeySize                   = 16
+	ReadMsgTimeout             = time.Second * 5
-	MaxRetry                     = 10
+	PaddingSize                = 16
-	RetryInterval                = time.Second * 30
+	AESKeySize                 = 16
-	PublicIPEchoTimeout          = time.Second * 1
+	MaxRetry                   = 10
-	NatTestTimeout               = time.Second * 5
+	Cone2ConeTCPPunchMaxRetry  = 1
-	ClientAPITimeout             = time.Second * 10
+	Cone2ConeUDPPunchMaxRetry  = 1
-	MaxDirectTry                 = 3
+	PublicIPEchoTimeout        = time.Second * 1
+	NatTestTimeout             = time.Second * 5
+	UDPReadTimeout             = time.Second * 5
+	ClientAPITimeout           = time.Second * 10
+	UnderlayConnectTimeout     = time.Second * 10
+	MaxDirectTry               = 3
+
+	// sdwan
+	ReadTunBuffSize = 1600
+	ReadTunBuffNum  = 10
 )
 
 // NATNone has public ip
@@ -170,8 +198,9 @@ const (
 const (
 	LinkModeUDPPunch = "udppunch"
 	LinkModeTCPPunch = "tcppunch"
-	LinkModeIPv4     = "ipv4" // for web
+	LinkModeIPv4     = "ipv4"     // for web
-	LinkModeIPv6     = "ipv6" // for web
+	LinkModeIntranet = "intranet" // for web
+	LinkModeIPv6     = "ipv6"     // for web
 	LinkModeTCP6     = "tcp6"
 	LinkModeTCP4     = "tcp4"
 	LinkModeUDP6     = "udp6"
@@ -183,6 +212,11 @@ const (
 	MsgQueryPeerInfoRsp
 )
 
+const (
+	MsgSDWANInfoReq = iota
+	MsgSDWANInfoRsp
+)
+
 func newMessage(mainType uint16, subType uint16, packet interface{}) ([]byte, error) {
 	data, err := json.Marshal(packet)
 	if err != nil {
@@ -203,7 +237,7 @@ func newMessage(mainType uint16, subType uint16, packet interface{}) ([]byte, er
 	return writeBytes, nil
 }
 
-func nodeNameToID(name string) uint64 {
+func NodeNameToID(name string) uint64 {
 	return crc64.Checksum([]byte(name), crc64.MakeTable(crc64.ISO))
 }
 
@@ -221,7 +255,11 @@ type PushConnectReq struct {
 	ID               uint64 `json:"id,omitempty"`
 	AppKey           uint64 `json:"appKey,omitempty"` // for underlay tcp
 	LinkMode         string `json:"linkMode,omitempty"`
-	IsUnderlayServer int    `json:"isServer,omitempty"` // Requset spec peer is server
+	IsUnderlayServer int    `json:"isServer,omitempty"`         // Requset spec peer is server
+	UnderlayProtocol string `json:"underlayProtocol,omitempty"` // quic or kcp, default quic
+}
+type PushDstNodeOnline struct {
+	Node string `json:"node,omitempty"`
 }
 type PushConnectRsp struct {
 	Error           int    `json:"error,omitempty"`
@@ -235,6 +273,7 @@ type PushConnectRsp struct {
 	ConeNatPort     int    `json:"coneNatPort,omitempty"` //it's not only cone, but also upnp or nat-pmp hole
 	FromIP          string `json:"fromIP,omitempty"`
 	ID              uint64 `json:"id,omitempty"`
+	PunchTs         uint64 `json:"punchts,omitempty"` // server timestamp
 	Version         string `json:"version,omitempty"`
 }
 type PushRsp struct {
@@ -243,12 +282,13 @@ type PushRsp struct {
 }
 
 type LoginRsp struct {
-	Error  int    `json:"error,omitempty"`
+	Error         int    `json:"error,omitempty"`
-	Detail string `json:"detail,omitempty"`
+	Detail        string `json:"detail,omitempty"`
-	User   string `json:"user,omitempty"`
+	User          string `json:"user,omitempty"`
-	Node   string `json:"node,omitempty"`
+	Node          string `json:"node,omitempty"`
-	Token  uint64 `json:"token,omitempty"`
+	Token         uint64 `json:"token,omitempty"`
-	Ts     int64  `json:"ts,omitempty"`
+	Ts            int64  `json:"ts,omitempty"`
+	LoginMaxDelay int    `json:"loginMaxDelay,omitempty"` // seconds
 }
 
 type NatDetectReq struct {
@@ -293,11 +333,13 @@ type RelayNodeRsp struct {
 }
 
 type AddRelayTunnelReq struct {
-	From       string `json:"from,omitempty"`
+	From          string `json:"from,omitempty"`
-	RelayName  string `json:"relayName,omitempty"`
+	RelayName     string `json:"relayName,omitempty"`
-	RelayToken uint64 `json:"relayToken,omitempty"`
+	RelayTunnelID uint64 `json:"relayTunnelID,omitempty"`
-	AppID      uint64 `json:"appID,omitempty"`  // deprecated
+	RelayToken    uint64 `json:"relayToken,omitempty"`
-	AppKey     uint64 `json:"appKey,omitempty"` // deprecated
+	RelayMode     string `json:"relayMode,omitempty"`
+	AppID         uint64 `json:"appID,omitempty"`  // deprecated
+	AppKey        uint64 `json:"appKey,omitempty"` // deprecated
 }
 
 type APPKeySync struct {
@@ -306,6 +348,7 @@ type APPKeySync struct {
 }
 
 type RelayHeartbeat struct {
+	From          string `json:"from,omitempty"`
 	RelayTunnelID uint64 `json:"relayTunnelID,omitempty"`
 	AppID         uint64 `json:"appID,omitempty"`
 }
@@ -341,9 +384,11 @@ type AppInfo struct {
 	AppName        string `json:"appName,omitempty"`
 	Error          string `json:"error,omitempty"`
 	Protocol       string `json:"protocol,omitempty"`
+	PunchPriority  int    `json:"punchPriority,omitempty"`
+	Whitelist      string `json:"whitelist,omitempty"`
 	SrcPort        int    `json:"srcPort,omitempty"`
 	Protocol0      string `json:"protocol0,omitempty"`
-	SrcPort0       int    `json:"srcPort0,omitempty"`
+	SrcPort0       int    `json:"srcPort0,omitempty"` // srcport+protocol is uneque, use as old app id
 	NatType        int    `json:"natType,omitempty"`
 	PeerNode       string `json:"peerNode,omitempty"`
 	DstPort        int    `json:"dstPort,omitempty"`
@@ -353,6 +398,7 @@ type AppInfo struct {
 	PeerIP         string `json:"peerIP,omitempty"`
 	ShareBandwidth int    `json:"shareBandWidth,omitempty"`
 	RelayNode      string `json:"relayNode,omitempty"`
+	SpecRelayNode  string `json:"specRelayNode,omitempty"`
 	RelayMode      string `json:"relayMode,omitempty"`
 	LinkMode       string `json:"linkMode,omitempty"`
 	Version        string `json:"version,omitempty"`
@@ -422,11 +468,161 @@ type QueryPeerInfoReq struct {
 	PeerNode string `json:"peerNode,omitempty"`
 }
 type QueryPeerInfoRsp struct {
+	PeerNode        string `json:"peerNode,omitempty"`
 	Online          int    `json:"online,omitempty"`
 	Version         string `json:"version,omitempty"`
 	NatType         int    `json:"natType,omitempty"`
 	IPv4            string `json:"IPv4,omitempty"`
+	LanIP           string `json:"lanIP,omitempty"`
 	HasIPv4         int    `json:"hasIPv4,omitempty"` // has public ipv4
 	IPv6            string `json:"IPv6,omitempty"`    // if public relay node, ipv6 not set
 	HasUPNPorNATPMP int    `json:"hasUPNPorNATPMP,omitempty"`
 }
+
+type SDWANNode struct {
+	Name     string `json:"name,omitempty"`
+	IP       string `json:"ip,omitempty"`
+	Resource string `json:"resource,omitempty"`
+	Enable   int32  `json:"enable,omitempty"`
+}
+
+type SDWANInfo struct {
+	ID            uint64 `json:"id,omitempty"`
+	Name          string `json:"name,omitempty"`
+	Gateway       string `json:"gateway,omitempty"`
+	Mode          string `json:"mode,omitempty"` // default: fullmesh; central
+	CentralNode   string `json:"centralNode,omitempty"`
+	ForceRelay    int32  `json:"forceRelay,omitempty"`
+	PunchPriority int32  `json:"punchPriority,omitempty"`
+	Enable        int32  `json:"enable,omitempty"`
+	Nodes         []SDWANNode
+}
+
+const (
+	SDWANModeFullmesh = "fullmesh"
+	SDWANModeCentral  = "central"
+)
+
+type ServerSideSaveMemApp struct {
+	From          string `json:"from,omitempty"`
+	Node          string `json:"node,omitempty"`          // for server side findtunnel, maybe relayNode
+	TunnelID      uint64 `json:"tunnelID,omitempty"`      // save in app.tunnel or app.relayTunnel
+	RelayTunnelID uint64 `json:"relayTunnelID,omitempty"` // rtid, if not 0 relay
+	RelayMode     string `json:"relayMode,omitempty"`
+	AppID         uint64 `json:"appID,omitempty"`
+}
+
+type CheckRemoteService struct {
+	Host string `json:"host,omitempty"`
+	Port uint32 `json:"port,omitempty"`
+}
+
+const rootCA = `-----BEGIN CERTIFICATE-----
+MIIDhTCCAm0CFHm0cd8dnGCbUW/OcS56jf0gvRk7MA0GCSqGSIb3DQEBCwUAMH4x
+CzAJBgNVBAYTAkNOMQswCQYDVQQIDAJHRDETMBEGA1UECgwKb3BlbnAycC5jbjET
+MBEGA1UECwwKb3BlbnAycC5jbjETMBEGA1UEAwwKb3BlbnAycC5jbjEjMCEGCSqG
+SIb3DQEJARYUb3BlbnAycC5jbkBnbWFpbC5jb20wIBcNMjMwODAxMDkwMjMwWhgP
+MjEyMzA3MDgwOTAyMzBaMH4xCzAJBgNVBAYTAkNOMQswCQYDVQQIDAJHRDETMBEG
+A1UECgwKb3BlbnAycC5jbjETMBEGA1UECwwKb3BlbnAycC5jbjETMBEGA1UEAwwK
+b3BlbnAycC5jbjEjMCEGCSqGSIb3DQEJARYUb3BlbnAycC5jbkBnbWFpbC5jb20w
+ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDWg8wPy5hBLUaY4WOXayKu
++magEz1LAY0krzXYSZaSCvGMwA0cervwAqgKfiiZEhho5UNA5iVOJ6bO1RL9H7Vp
+4HuW9BttDU/NQHguD8pyqx06Kaosz5LRw8USz1BCWWFdmi8Mv4I0omtd7m6lbWnY
+nrjQKLYPahPW481jUfJPqR6wUTnBuBMr2ZAGqmFR4Lhqs9B1P9GeBfDWNwVApJUC
+VEhbElukRJxdUvWeJ5+HMENKQcHCTTgmQbmDLMobHXs3Xf7fT9qC76wOe9LFHI6L
+dAww9gryQhxWauQl1NO8aGJTFu+3wgnKBdTMJmF/1iuZYXJOCR1solwqU1hCgBsj
+AgMBAAEwDQYJKoZIhvcNAQELBQADggEBADp153YNVN8p6/3PLnXxHBDeDViAfeQd
+VJmy8eH1LTq/xtUY71HGSpL7iIBNoQdDTHfsg3c6ZANBCxbO/7AhFAzPt1aK8eHy
+XuEiW0Z6R8np1Khh3alCOfD15tKcjok//Wxisbz+YItlbDus/eWRbLGB3HGrzn4l
+GB18jw+G7o4U3rGX8agHqVGQEd06gk1ZaprASpTGwSsv4A5ehosjT1d7re8Z5eD4
+RVtXS+DplMClQ5QSlv3StwcWOsjyiAimNfLEU5xoEfq17yOJUTU1OTL4YOt16QUc
+C1tnzFr3k/ioqFR7cnyzNrbjlfPOmO9l2WReEbMP3bvaSHm6EcpJKS8=
+-----END CERTIFICATE-----`
+
+const ISRGRootX1 = `-----BEGIN CERTIFICATE-----
+MIIEJjCCAw6gAwIBAgISAztStWq026ej0RCsk3ErbUdPMA0GCSqGSIb3DQEBCwUA
+MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
+EwJSMzAeFw0yMzA4MDQwODUyMjlaFw0yMzExMDIwODUyMjhaMBcxFTATBgNVBAMM
+DCoub3BlbnAycC5jbjBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABPRdkgLV2FA+
+3g/GjcA9UcfDfIFYgofSTNbOCQFIiQVMXrTgAToF1/tWaS2LOuysZcCX6OE7SCeG
+lQ+0g+L2qvujggIaMIICFjAOBgNVHQ8BAf8EBAMCB4AwHQYDVR0lBBYwFAYIKwYB
+BQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFIdL5LNQC+X4
+8r6u+3NlM238Vmk5MB8GA1UdIwQYMBaAFBQusxe3WFbLrlAJQOYfr52LFMLGMFUG
+CCsGAQUFBwEBBEkwRzAhBggrBgEFBQcwAYYVaHR0cDovL3IzLm8ubGVuY3Iub3Jn
+MCIGCCsGAQUFBzAChhZodHRwOi8vcjMuaS5sZW5jci5vcmcvMCMGA1UdEQQcMBqC
+DCoub3BlbnAycC5jboIKb3BlbnAycC5jbjATBgNVHSAEDDAKMAgGBmeBDAECATCC
+AQQGCisGAQQB1nkCBAIEgfUEgfIA8AB2AHoyjFTYty22IOo44FIe6YQWcDIThU07
+0ivBOlejUutSAAABib/2fCgAAAQDAEcwRQIhAJzf9XNe0cu9CNYLLqtDCZZMqI6u
+qsHrnnXcFQW23ioZAiAgwKp5DwZw9RmF19KOjD6lYJfTxc+anJUuWAlMwu1HYQB2
+AK33vvp8/xDIi509nB4+GGq0Zyldz7EMJMqFhjTr3IKKAAABib/2fEEAAAQDAEcw
+RQIgKeI7DopyzFXPdRQZKZrHVqfXQ8OipvlKXd5xRnKFjH4CIQDMM+TU+LOux8xK
+1NlTiSs9DhQI/eU3ZXKxSQAqF50RnTANBgkqhkiG9w0BAQsFAAOCAQEATqZ+H2NT
+cv4FzArD/Krlnur1OTitvpubRWM+ClB9Cr6pvPVB7Dp0/ALxu35ZmCtrzdJWTfmp
+lHxU4nPXRPVjuPRNXooSyH//KTfHyf32919PQOi/qc/QEAuIzkGLJg0dIPKLxaNK
+CiTWU+2iAYSHBgCWulfLX/RYNbBZQ9w0xIm3XhuMjCF/omG8ofuz1DmiRVR+17JA
+nuDXQkxm7KhmbxSA4PsLwzvIWA8Wk44ZK7uncgRY3WIUXcVRELSFA5LuH67TOwag
+al6iG56KW1N2Yy9YmeG27SYvHZYkjmuJ8NEy7Ku+Mi6gwO4hs0CYr2wtUacPfjKF
+aYTGWSt6Pt8kmw==
+-----END CERTIFICATE-----
+-----BEGIN CERTIFICATE-----
+MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
+TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
+cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
+WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
+RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
+AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
+R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
+sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
+NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
+Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
+/kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
+AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
+Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
+FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
+AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
+Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
+gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
+PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
+ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
+CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
+lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
+avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
+yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
+yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
+hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
+HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
+MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
+nLRbwHOoq7hHwg==
+-----END CERTIFICATE-----
+-----BEGIN CERTIFICATE-----
+MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw
+TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
+cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4
+WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu
+ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY
+MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc
+h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+
+0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U
+A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW
+T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH
+B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC
+B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv
+KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn
+OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn
+jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw
+qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI
+rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV
+HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq
+hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL
+ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ
+3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK
+NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5
+ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur
+TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC
+jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc
+oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq
+4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA
+mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d
+emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=
+-----END CERTIFICATE-----
+`

core/sdwan.go

@@ -0,0 +1,292 @@
+package openp2p
+
+import (
+	"encoding/binary"
+	"encoding/json"
+	"fmt"
+	"net"
+	"runtime"
+	"strings"
+	"sync"
+	"time"
+)
+
+type PacketHeader struct {
+	version int
+	// src     uint32
+	// prot    uint8
+	protocol byte
+	dst      uint32
+	port     uint16
+}
+
+func parseHeader(b []byte, h *PacketHeader) error {
+	if len(b) < 20 {
+		return fmt.Errorf("small packet")
+	}
+	h.version = int(b[0] >> 4)
+	h.protocol = byte(b[9])
+	if h.version == 4 {
+		h.dst = binary.BigEndian.Uint32(b[16:20])
+	} else if h.version != 6 {
+		return fmt.Errorf("unknown version in ip header:%d", h.version)
+	}
+	if h.protocol == 6 || h.protocol == 17 { // TCP or UDP
+		h.port = binary.BigEndian.Uint16(b[22:24])
+	}
+	return nil
+}
+
+type sdwanNode struct {
+	name string
+	id   uint64
+}
+
+type p2pSDWAN struct {
+	nodeName      string
+	tun           *optun
+	sysRoute      sync.Map // ip:sdwanNode
+	subnet        *net.IPNet
+	gateway       net.IP
+	virtualIP     *net.IPNet
+	internalRoute *IPTree
+}
+
+func (s *p2pSDWAN) init(name string) error {
+	if gConf.getSDWAN().Gateway == "" {
+		gLog.Println(LvDEBUG, "not in sdwan clear all ")
+	}
+	if s.internalRoute == nil {
+		s.internalRoute = NewIPTree("")
+	}
+
+	s.nodeName = name
+	s.gateway, s.subnet, _ = net.ParseCIDR(gConf.getSDWAN().Gateway)
+	for _, node := range gConf.getDelNodes() {
+		gLog.Println(LvDEBUG, "deal deleted node: ", node.Name)
+		delRoute(node.IP, s.gateway.String())
+		s.internalRoute.Del(node.IP, node.IP)
+		ipNum, _ := inetAtoN(node.IP)
+		s.sysRoute.Delete(ipNum)
+		gConf.delete(AppConfig{SrcPort: 0, PeerNode: node.Name})
+		GNetwork.DeleteApp(AppConfig{SrcPort: 0, PeerNode: node.Name})
+		arr := strings.Split(node.Resource, ",")
+		for _, r := range arr {
+			_, ipnet, err := net.ParseCIDR(r)
+			if err != nil {
+				// fmt.Println("Error parsing CIDR:", err)
+				continue
+			}
+			if ipnet.Contains(net.ParseIP(gConf.Network.localIP)) { // local ip and resource in the same lan
+				continue
+			}
+			minIP := ipnet.IP
+			maxIP := make(net.IP, len(minIP))
+			copy(maxIP, minIP)
+			for i := range minIP {
+				maxIP[i] = minIP[i] | ^ipnet.Mask[i]
+			}
+			s.internalRoute.Del(minIP.String(), maxIP.String())
+			delRoute(ipnet.String(), s.gateway.String())
+		}
+	}
+	for _, node := range gConf.getAddNodes() {
+		gLog.Println(LvDEBUG, "deal add node: ", node.Name)
+		ipNet := &net.IPNet{
+			IP:   net.ParseIP(node.IP),
+			Mask: s.subnet.Mask,
+		}
+		if node.Name == s.nodeName {
+			s.virtualIP = ipNet
+			gLog.Println(LvINFO, "start tun ", ipNet.String())
+			err := s.StartTun()
+			if err != nil {
+				gLog.Println(LvERROR, "start tun error:", err)
+				return err
+			}
+			gLog.Println(LvINFO, "start tun ok")
+			allowTunForward()
+			addRoute(s.subnet.String(), s.gateway.String(), s.tun.tunName)
+			// addRoute("255.255.255.255/32", s.gateway.String(), s.tun.tunName) // for broadcast
+			// addRoute("224.0.0.0/4", s.gateway.String(), s.tun.tunName)        // for multicast
+			initSNATRule(s.subnet.String()) // for network resource
+			continue
+		}
+		ip, err := inetAtoN(ipNet.String())
+		if err != nil {
+			return err
+		}
+		s.sysRoute.Store(ip, &sdwanNode{name: node.Name, id: NodeNameToID(node.Name)})
+		s.internalRoute.AddIntIP(ip, ip, &sdwanNode{name: node.Name, id: NodeNameToID(node.Name)})
+	}
+	for _, node := range gConf.getAddNodes() {
+		if node.Name == s.nodeName { // not deal resource itself
+			continue
+		}
+		if len(node.Resource) > 0 {
+			gLog.Printf(LvINFO, "deal add node: %s resource: %s", node.Name, node.Resource)
+			arr := strings.Split(node.Resource, ",")
+			for _, r := range arr {
+				// add internal route
+				_, ipnet, err := net.ParseCIDR(r)
+				if err != nil {
+					fmt.Println("Error parsing CIDR:", err)
+					continue
+				}
+				if ipnet.Contains(net.ParseIP(gConf.Network.localIP)) { // local ip and resource in the same lan
+					continue
+				}
+				minIP := ipnet.IP
+				maxIP := make(net.IP, len(minIP))
+				copy(maxIP, minIP)
+				for i := range minIP {
+					maxIP[i] = minIP[i] | ^ipnet.Mask[i]
+				}
+				s.internalRoute.Add(minIP.String(), maxIP.String(), &sdwanNode{name: node.Name, id: NodeNameToID(node.Name)})
+				// add sys route
+				addRoute(ipnet.String(), s.gateway.String(), s.tun.tunName)
+			}
+		}
+	}
+	gConf.retryAllMemApp()
+	gLog.Printf(LvINFO, "sdwan init ok")
+	return nil
+}
+
+func (s *p2pSDWAN) run() {
+	s.sysRoute.Range(func(key, value interface{}) bool {
+		node := value.(*sdwanNode)
+		GNetwork.ConnectNode(node.name)
+		return true
+	})
+}
+
+func (s *p2pSDWAN) readNodeLoop() {
+	gLog.Printf(LvDEBUG, "sdwan readNodeLoop start")
+	defer gLog.Printf(LvDEBUG, "sdwan readNodeLoop end")
+	writeBuff := make([][]byte, 1)
+	for {
+		nd := GNetwork.ReadNode(time.Second * 10) // TODO: read multi packet
+		if nd == nil {
+			gLog.Printf(LvDev, "waiting for node data")
+			continue
+		}
+		head := PacketHeader{}
+		parseHeader(nd.Data, &head)
+		gLog.Printf(LvDev, "write tun dst ip=%s,len=%d", net.IP{byte(head.dst >> 24), byte(head.dst >> 16), byte(head.dst >> 8), byte(head.dst)}.String(), len(nd.Data))
+		if PIHeaderSize == 0 {
+			writeBuff[0] = nd.Data
+		} else {
+			writeBuff[0] = make([]byte, PIHeaderSize+len(nd.Data))
+			copy(writeBuff[0][PIHeaderSize:], nd.Data)
+		}
+
+		len, err := s.tun.Write(writeBuff, PIHeaderSize)
+		if err != nil {
+			gLog.Printf(LvDEBUG, "write tun dst ip=%s,len=%d,error:%s", net.IP{byte(head.dst >> 24), byte(head.dst >> 16), byte(head.dst >> 8), byte(head.dst)}.String(), len, err)
+		}
+	}
+}
+
+func isBroadcastOrMulticast(ipUint32 uint32, subnet *net.IPNet) bool {
+	// return ipUint32 == 0xffffffff || (byte(ipUint32) == 0xff) || (ipUint32>>28 == 0xe)
+	return ipUint32 == 0xffffffff || (ipUint32>>28 == 0xe) // 225.255.255.255/32, 224.0.0.0/4
+}
+
+func (s *p2pSDWAN) routeTunPacket(p []byte, head *PacketHeader) {
+	var node *sdwanNode
+	// v, ok := s.routes.Load(ih.dst)
+	v, ok := s.internalRoute.Load(head.dst)
+	if !ok || v == nil {
+		if isBroadcastOrMulticast(head.dst, s.subnet) {
+			gLog.Printf(LvDev, "multicast ip=%s", net.IP{byte(head.dst >> 24), byte(head.dst >> 16), byte(head.dst >> 8), byte(head.dst)}.String())
+			GNetwork.WriteBroadcast(p)
+		}
+		return
+	} else {
+		node = v.(*sdwanNode)
+	}
+
+	err := GNetwork.WriteNode(node.id, p)
+	if err != nil {
+		gLog.Printf(LvDev, "write packet to %s fail: %s", node.name, err)
+	}
+}
+
+func (s *p2pSDWAN) readTunLoop() {
+	gLog.Printf(LvDEBUG, "sdwan readTunLoop start")
+	defer gLog.Printf(LvDEBUG, "sdwan readTunLoop end")
+	readBuff := make([][]byte, ReadTunBuffNum)
+	for i := 0; i < ReadTunBuffNum; i++ {
+		readBuff[i] = make([]byte, ReadTunBuffSize+PIHeaderSize)
+	}
+	readBuffSize := make([]int, ReadTunBuffNum)
+	ih := PacketHeader{}
+	for {
+		n, err := s.tun.Read(readBuff, readBuffSize, PIHeaderSize)
+		if err != nil {
+			gLog.Printf(LvERROR, "read tun fail: ", err)
+			return
+		}
+		for i := 0; i < n; i++ {
+			if readBuffSize[i] > ReadTunBuffSize {
+				gLog.Printf(LvERROR, "read tun overflow: len=", readBuffSize[i])
+				continue
+			}
+			parseHeader(readBuff[i][PIHeaderSize:readBuffSize[i]+PIHeaderSize], &ih)
+			gLog.Printf(LvDev, "read tun dst ip=%s,len=%d", net.IP{byte(ih.dst >> 24), byte(ih.dst >> 16), byte(ih.dst >> 8), byte(ih.dst)}.String(), readBuffSize[0])
+			s.routeTunPacket(readBuff[i][PIHeaderSize:readBuffSize[i]+PIHeaderSize], &ih)
+		}
+	}
+}
+
+func (s *p2pSDWAN) StartTun() error {
+	sdwan := gConf.getSDWAN()
+	if s.tun == nil {
+		tun := &optun{}
+		err := tun.Start(s.virtualIP.String(), &sdwan)
+		if err != nil {
+			gLog.Println(LvERROR, "open tun fail:", err)
+			return err
+		}
+		s.tun = tun
+		go s.readTunLoop()
+		go s.readNodeLoop() // multi-thread read will cause packets out of order, resulting in slower speeds
+	}
+	err := setTunAddr(s.tun.tunName, s.virtualIP.String(), sdwan.Gateway, s.tun.dev)
+	if err != nil {
+		gLog.Printf(LvERROR, "setTunAddr error:%s,%s,%s,%s", err, s.tun.tunName, s.virtualIP.String(), sdwan.Gateway)
+		return err
+	}
+	return nil
+}
+
+func handleSDWAN(subType uint16, msg []byte) error {
+	gLog.Printf(LvDEBUG, "handle sdwan msg type:%d", subType)
+	var err error
+	switch subType {
+	case MsgSDWANInfoRsp:
+		rsp := SDWANInfo{}
+		if err = json.Unmarshal(msg[openP2PHeaderSize:], &rsp); err != nil {
+			return ErrMsgFormat
+		}
+		gLog.Println(LvINFO, "sdwan init:", prettyJson(rsp))
+		if runtime.GOOS == "android" {
+			AndroidSDWANConfig <- msg[openP2PHeaderSize:]
+		}
+		// GNetwork.sdwan.detail = &rsp
+		gConf.setSDWAN(rsp)
+		err = GNetwork.sdwan.init(gConf.Network.Node)
+		if err != nil {
+			gLog.Println(LvERROR, "sdwan init fail: ", err)
+			if GNetwork.sdwan.tun != nil {
+				GNetwork.sdwan.tun.Stop()
+				GNetwork.sdwan.tun = nil
+				return err
+			}
+		}
+		go GNetwork.sdwan.run()
+	default:
+	}
+	return err
+}

core/speedlimiter.go

@@ -0,0 +1,50 @@
+package openp2p
+
+import (
+	"sync"
+	"time"
+)
+
+// SpeedLimiter ...
+type SpeedLimiter struct {
+	lastUpdate time.Time
+	speed      int // per second
+	precision  int // seconds
+	freeCap    int
+	maxFreeCap int
+	mtx        sync.Mutex
+}
+
+func newSpeedLimiter(speed int, precision int) *SpeedLimiter {
+	return &SpeedLimiter{
+		speed:      speed,
+		precision:  precision,
+		lastUpdate: time.Now(),
+		maxFreeCap: speed * precision,
+		freeCap:    speed * precision,
+	}
+}
+
+// Add ...
+func (sl *SpeedLimiter) Add(increment int, wait bool) bool {
+	if sl.speed <= 0 {
+		return true
+	}
+	sl.mtx.Lock()
+	defer sl.mtx.Unlock()
+	sl.freeCap += int(time.Since(sl.lastUpdate) * time.Duration(sl.speed) / time.Second)
+	if sl.freeCap > sl.maxFreeCap {
+		sl.freeCap = sl.maxFreeCap
+	}
+	if !wait && sl.freeCap < increment {
+		return false
+	}
+	sl.freeCap -= increment
+	sl.lastUpdate = time.Now()
+	if sl.freeCap < 0 {
+		// sleep for the overflow
+		// fmt.Println("sleep ", time.Millisecond*time.Duration(-sl.freeCap*100)/time.Duration(sl.speed))
+		time.Sleep(time.Millisecond * time.Duration(-sl.freeCap*1000) / time.Duration(sl.speed)) // sleep ms
+	}
+	return true
+}

core/speedlimiter_test.go

@@ -0,0 +1,59 @@
+package openp2p
+
+import (
+	"testing"
+	"time"
+)
+
+func TestBandwidth(t *testing.T) {
+	speed := 10 * 1024 * 1024 / 8 // 10mbps
+	speedl := newSpeedLimiter(speed, 1)
+	oneBuffSize := 4096
+	writeNum := 5000
+	expectTime := oneBuffSize * writeNum / speed
+	startTs := time.Now()
+	for i := 0; i < writeNum; i++ {
+		speedl.Add(oneBuffSize, true)
+	}
+	t.Logf("cost %ds, expect %ds", time.Since(startTs)/time.Second, expectTime)
+	if time.Since(startTs) > time.Duration(expectTime+1)*time.Second || time.Since(startTs) < time.Duration(expectTime-1)*time.Second {
+		t.Error("error")
+	}
+}
+
+func TestSymmetric(t *testing.T) {
+	speed := 20000 / 180
+	speedl := newSpeedLimiter(speed, 180)
+	oneBuffSize := 300
+	writeNum := 70
+	expectTime := (oneBuffSize*writeNum - 20000) / speed
+	t.Logf("expect %ds", expectTime)
+	startTs := time.Now()
+	for i := 0; i < writeNum; i++ {
+		speedl.Add(oneBuffSize, true)
+	}
+	t.Logf("cost %ds, expect %ds", time.Since(startTs)/time.Second, expectTime)
+	if time.Since(startTs) > time.Duration(expectTime+1)*time.Second || time.Since(startTs) < time.Duration(expectTime-1)*time.Second {
+		t.Error("error")
+	}
+}
+
+func TestSymmetric2(t *testing.T) {
+	speed := 30000 / 180
+	speedl := newSpeedLimiter(speed, 180)
+	oneBuffSize := 800
+	writeNum := 40
+	expectTime := (oneBuffSize*writeNum - 30000) / speed
+	startTs := time.Now()
+	for i := 0; i < writeNum; {
+		if speedl.Add(oneBuffSize, true) {
+			i++
+		} else {
+			time.Sleep(time.Millisecond)
+		}
+	}
+	t.Logf("cost %ds, expect %ds", time.Since(startTs)/time.Second, expectTime)
+	if time.Since(startTs) > time.Duration(expectTime+1)*time.Second || time.Since(startTs) < time.Duration(expectTime-1)*time.Second {
+		t.Error("error")
+	}
+}

core/totp.go

@@ -1,35 +0,0 @@
-// Time-based One-time Password
-package openp2p
-
-import (
-	"crypto/hmac"
-	"crypto/sha256"
-	"encoding/binary"
-)
-
-const TOTPStep = 30 // 30s
-func GenTOTP(token uint64, ts int64) uint64 {
-	step := ts / TOTPStep
-	tbuff := make([]byte, 8)
-	binary.LittleEndian.PutUint64(tbuff, token)
-	mac := hmac.New(sha256.New, tbuff)
-	b := make([]byte, 8)
-	binary.LittleEndian.PutUint64(b, uint64(step))
-	mac.Write(b)
-	num := binary.LittleEndian.Uint64(mac.Sum(nil)[:8])
-	// fmt.Printf("%x\n", mac.Sum(nil))
-	return num
-}
-
-func VerifyTOTP(code uint64, token uint64, ts int64) bool {
-	if code == 0 {
-		return false
-	}
-	if code == token {
-		return true
-	}
-	if code == GenTOTP(token, ts) || code == GenTOTP(token, ts-TOTPStep) || code == GenTOTP(token, ts+TOTPStep) {
-		return true
-	}
-	return false
-}

core/totp_test.go

@@ -1,36 +0,0 @@
-// Time-based One-time Password
-package openp2p
-
-import (
-	"testing"
-	"time"
-)
-
-func TestTOTP(t *testing.T) {
-	for i := 0; i < 20; i++ {
-		ts := time.Now().Unix()
-		code := GenTOTP(13666999958022769123, ts)
-		t.Log(code)
-		if !VerifyTOTP(code, 13666999958022769123, ts) {
-			t.Error("TOTP error")
-		}
-		if !VerifyTOTP(code, 13666999958022769123, ts-10) {
-			t.Error("TOTP error")
-		}
-		if !VerifyTOTP(code, 13666999958022769123, ts+10) {
-			t.Error("TOTP error")
-		}
-		if VerifyTOTP(code, 13666999958022769123, ts+60) {
-			t.Error("TOTP error")
-		}
-		if VerifyTOTP(code, 13666999958022769124, ts+1) {
-			t.Error("TOTP error")
-		}
-		if VerifyTOTP(code, 13666999958022769125, ts+1) {
-			t.Error("TOTP error")
-		}
-		time.Sleep(time.Second)
-		t.Log("round", i, " ", ts, " test ok")
-	}
-
-}

core/udp.go

@@ -18,25 +18,24 @@ func UDPWrite(conn *net.UDPConn, dst net.Addr, mainType uint16, subType uint16,
 	return conn.WriteTo(msg, dst)
 }
 
-func UDPRead(conn *net.UDPConn, timeout int) (ra net.Addr, head *openP2PHeader, result []byte, len int, err error) {
+func UDPRead(conn *net.UDPConn, timeout time.Duration) (ra net.Addr, head *openP2PHeader, buff []byte, length int, err error) {
 	if timeout > 0 {
-		deadline := time.Now().Add(time.Millisecond * time.Duration(timeout))
+		err = conn.SetReadDeadline(time.Now().Add(timeout))
-		err = conn.SetReadDeadline(deadline)
 		if err != nil {
 			gLog.Println(LvERROR, "SetReadDeadline error")
 			return nil, nil, nil, 0, err
 		}
 	}
 
-	result = make([]byte, 1024)
+	buff = make([]byte, 1024)
-	len, ra, err = conn.ReadFrom(result)
+	length, ra, err = conn.ReadFrom(buff)
 	if err != nil {
 		// gLog.Println(LevelDEBUG, "ReadFrom error")
 		return nil, nil, nil, 0, err
 	}
 	head = &openP2PHeader{}
-	err = binary.Read(bytes.NewReader(result[:openP2PHeaderSize]), binary.LittleEndian, head)
+	err = binary.Read(bytes.NewReader(buff[:openP2PHeaderSize]), binary.LittleEndian, head)
-	if err != nil {
+	if err != nil || head.DataLen > uint32(len(buff)-openP2PHeaderSize) {
 		gLog.Println(LvERROR, "parse p2pheader error:", err)
 		return nil, nil, nil, 0, err
 	}

core/underlay.go

@@ -1,16 +1,65 @@
 package openp2p
 
 import (
+	"io"
 	"time"
 )
 
 type underlay interface {
+	Read([]byte) (int, error)
+	Write([]byte) (int, error)
 	ReadBuffer() (*openP2PHeader, []byte, error)
 	WriteBytes(uint16, uint16, []byte) error
 	WriteBuffer([]byte) error
 	WriteMessage(uint16, uint16, interface{}) error
 	Close() error
+	WLock()
+	WUnlock()
 	SetReadDeadline(t time.Time) error
 	SetWriteDeadline(t time.Time) error
 	Protocol() string
 }
+
+func DefaultReadBuffer(ul underlay) (*openP2PHeader, []byte, error) {
+	headBuf := make([]byte, openP2PHeaderSize)
+	_, err := io.ReadFull(ul, headBuf)
+	if err != nil {
+		return nil, nil, err
+	}
+	head, err := decodeHeader(headBuf)
+	if err != nil {
+		return nil, nil, err
+	}
+	dataBuf := make([]byte, head.DataLen)
+	_, err = io.ReadFull(ul, dataBuf)
+	return head, dataBuf, err
+}
+
+func DefaultWriteBytes(ul underlay, mainType, subType uint16, data []byte) error {
+	writeBytes := append(encodeHeader(mainType, subType, uint32(len(data))), data...)
+	ul.SetWriteDeadline(time.Now().Add(TunnelHeartbeatTime / 2))
+	ul.WLock()
+	_, err := ul.Write(writeBytes)
+	ul.WUnlock()
+	return err
+}
+
+func DefaultWriteBuffer(ul underlay, data []byte) error {
+	ul.SetWriteDeadline(time.Now().Add(TunnelHeartbeatTime / 2))
+	ul.WLock()
+	_, err := ul.Write(data)
+	ul.WUnlock()
+	return err
+}
+
+func DefaultWriteMessage(ul underlay, mainType uint16, subType uint16, packet interface{}) error {
+	writeBytes, err := newMessage(mainType, subType, packet)
+	if err != nil {
+		return err
+	}
+	ul.SetWriteDeadline(time.Now().Add(TunnelHeartbeatTime / 2))
+	ul.WLock()
+	_, err = ul.Write(writeBytes)
+	ul.WUnlock()
+	return err
+}

core/underlay_kcp.go

@@ -0,0 +1,94 @@
+package openp2p
+
+import (
+	"fmt"
+	"net"
+	"sync"
+	"time"
+
+	"github.com/xtaci/kcp-go/v5"
+)
+
+type underlayKCP struct {
+	listener *kcp.Listener
+	writeMtx *sync.Mutex
+	*kcp.UDPSession
+}
+
+func (conn *underlayKCP) Protocol() string {
+	return "kcp"
+}
+
+func (conn *underlayKCP) ReadBuffer() (*openP2PHeader, []byte, error) {
+	return DefaultReadBuffer(conn)
+}
+
+func (conn *underlayKCP) WriteBytes(mainType uint16, subType uint16, data []byte) error {
+	return DefaultWriteBytes(conn, mainType, subType, data)
+}
+
+func (conn *underlayKCP) WriteBuffer(data []byte) error {
+	return DefaultWriteBuffer(conn, data)
+}
+
+func (conn *underlayKCP) WriteMessage(mainType uint16, subType uint16, packet interface{}) error {
+	return DefaultWriteMessage(conn, mainType, subType, packet)
+}
+
+func (conn *underlayKCP) Close() error {
+	conn.UDPSession.Close()
+	return nil
+}
+func (conn *underlayKCP) WLock() {
+	conn.writeMtx.Lock()
+}
+func (conn *underlayKCP) WUnlock() {
+	conn.writeMtx.Unlock()
+}
+func (conn *underlayKCP) CloseListener() {
+	if conn.listener != nil {
+		conn.listener.Close()
+	}
+}
+
+func (conn *underlayKCP) Accept() error {
+	kConn, err := conn.listener.AcceptKCP()
+	if err != nil {
+		conn.listener.Close()
+		return err
+	}
+	kConn.SetNoDelay(0, 40, 0, 0)
+	kConn.SetWindowSize(512, 512)
+	kConn.SetWriteBuffer(1024 * 128)
+	kConn.SetReadBuffer(1024 * 128)
+	conn.UDPSession = kConn
+	return nil
+}
+
+func listenKCP(addr string, idleTimeout time.Duration) (*underlayKCP, error) {
+	gLog.Println(LvDEBUG, "kcp listen on ", addr)
+	listener, err := kcp.ListenWithOptions(addr, nil, 0, 0)
+	if err != nil {
+		return nil, fmt.Errorf("quic.ListenAddr error:%s", err)
+	}
+	ul := &underlayKCP{listener: listener, writeMtx: &sync.Mutex{}}
+	err = ul.Accept()
+	if err != nil {
+		ul.CloseListener()
+		return nil, fmt.Errorf("accept KCP error:%s", err)
+	}
+	return ul, nil
+}
+
+func dialKCP(conn *net.UDPConn, remoteAddr *net.UDPAddr, idleTimeout time.Duration) (*underlayKCP, error) {
+	kConn, err := kcp.NewConn(remoteAddr.String(), nil, 0, 0, conn)
+	if err != nil {
+		return nil, fmt.Errorf("quic.DialContext error:%s", err)
+	}
+	kConn.SetNoDelay(0, 40, 0, 0)
+	kConn.SetWindowSize(512, 512)
+	kConn.SetWriteBuffer(1024 * 128)
+	kConn.SetReadBuffer(1024 * 128)
+	ul := &underlayKCP{nil, &sync.Mutex{}, kConn}
+	return ul, nil
+}

core/underlay_quic.go

@@ -6,19 +6,17 @@ import (
 	"crypto/rsa"
 	"crypto/tls"
 	"crypto/x509"
-	"encoding/json"
 	"encoding/pem"
 	"fmt"
-	"io"
 	"math/big"
 	"net"
 	"sync"
 	"time"
 
-	"github.com/lucas-clemente/quic-go"
+	"github.com/quic-go/quic-go"
 )
 
-//quic.DialContext do not support version 44,disable it
+// quic.DialContext do not support version 44,disable it
 var quicVersion []quic.VersionNumber
 
 type underlayQUIC struct {
@@ -33,53 +31,33 @@ func (conn *underlayQUIC) Protocol() string {
 }
 
 func (conn *underlayQUIC) ReadBuffer() (*openP2PHeader, []byte, error) {
-	headBuf := make([]byte, openP2PHeaderSize)
+	return DefaultReadBuffer(conn)
-	_, err := io.ReadFull(conn, headBuf)
-	if err != nil {
-		return nil, nil, err
-	}
-	head, err := decodeHeader(headBuf)
-	if err != nil {
-		return nil, nil, err
-	}
-	dataBuf := make([]byte, head.DataLen)
-	_, err = io.ReadFull(conn, dataBuf)
-	return head, dataBuf, err
 }
 
 func (conn *underlayQUIC) WriteBytes(mainType uint16, subType uint16, data []byte) error {
-	writeBytes := append(encodeHeader(mainType, subType, uint32(len(data))), data...)
+	return DefaultWriteBytes(conn, mainType, subType, data)
-	conn.writeMtx.Lock()
-	_, err := conn.Write(writeBytes)
-	conn.writeMtx.Unlock()
-	return err
 }
 
 func (conn *underlayQUIC) WriteBuffer(data []byte) error {
-	conn.writeMtx.Lock()
+	return DefaultWriteBuffer(conn, data)
-	_, err := conn.Write(data)
-	conn.writeMtx.Unlock()
-	return err
 }
 
 func (conn *underlayQUIC) WriteMessage(mainType uint16, subType uint16, packet interface{}) error {
-	// TODO: call newMessage
+	return DefaultWriteMessage(conn, mainType, subType, packet)
-	data, err := json.Marshal(packet)
-	if err != nil {
-		return err
-	}
-	writeBytes := append(encodeHeader(mainType, subType, uint32(len(data))), data...)
-	conn.writeMtx.Lock()
-	_, err = conn.Write(writeBytes)
-	conn.writeMtx.Unlock()
-	return err
 }
 
 func (conn *underlayQUIC) Close() error {
 	conn.Stream.CancelRead(1)
 	conn.Connection.CloseWithError(0, "")
+	conn.CloseListener()
 	return nil
 }
+func (conn *underlayQUIC) WLock() {
+	conn.writeMtx.Lock()
+}
+func (conn *underlayQUIC) WUnlock() {
+	conn.writeMtx.Unlock()
+}
 func (conn *underlayQUIC) CloseListener() {
 	if conn.listener != nil {
 		conn.listener.Close()
@@ -87,7 +65,7 @@ func (conn *underlayQUIC) CloseListener() {
 }
 
 func (conn *underlayQUIC) Accept() error {
-	ctx, cancel := context.WithTimeout(context.Background(), time.Second*10)
+	ctx, cancel := context.WithTimeout(context.Background(), UnderlayConnectTimeout)
 	defer cancel()
 	sess, err := conn.listener.Accept(ctx)
 	if err != nil {
@@ -109,7 +87,13 @@ func listenQuic(addr string, idleTimeout time.Duration) (*underlayQUIC, error) {
 	if err != nil {
 		return nil, fmt.Errorf("quic.ListenAddr error:%s", err)
 	}
-	return &underlayQUIC{listener: listener, writeMtx: &sync.Mutex{}}, nil
+	ul := &underlayQUIC{listener: listener, writeMtx: &sync.Mutex{}}
+	err = ul.Accept()
+	if err != nil {
+		ul.CloseListener()
+		return nil, fmt.Errorf("accept quic error:%s", err)
+	}
+	return ul, nil
 }
 
 func dialQuic(conn *net.UDPConn, remoteAddr *net.UDPAddr, idleTimeout time.Duration) (*underlayQUIC, error) {

core/underlay_tcp.go

@@ -1,9 +1,8 @@
 package openp2p
 
 import (
-	"encoding/json"
+	"encoding/binary"
 	"fmt"
-	"io"
 	"net"
 	"sync"
 	"time"
@@ -14,6 +13,7 @@ import (
 type underlayTCP struct {
 	writeMtx *sync.Mutex
 	net.Conn
+	connectTime time.Time
 }
 
 func (conn *underlayTCP) Protocol() string {
@@ -21,88 +21,115 @@ func (conn *underlayTCP) Protocol() string {
 }
 
 func (conn *underlayTCP) ReadBuffer() (*openP2PHeader, []byte, error) {
-	headBuf := make([]byte, openP2PHeaderSize)
+	return DefaultReadBuffer(conn)
-	_, err := io.ReadFull(conn, headBuf)
-	if err != nil {
-		return nil, nil, err
-	}
-	head, err := decodeHeader(headBuf)
-	if err != nil {
-		return nil, nil, err
-	}
-	dataBuf := make([]byte, head.DataLen)
-	_, err = io.ReadFull(conn, dataBuf)
-	return head, dataBuf, err
 }
 
 func (conn *underlayTCP) WriteBytes(mainType uint16, subType uint16, data []byte) error {
-	writeBytes := append(encodeHeader(mainType, subType, uint32(len(data))), data...)
+	return DefaultWriteBytes(conn, mainType, subType, data)
-	conn.writeMtx.Lock()
-	_, err := conn.Write(writeBytes)
-	conn.writeMtx.Unlock()
-	return err
 }
 
 func (conn *underlayTCP) WriteBuffer(data []byte) error {
-	conn.writeMtx.Lock()
+	return DefaultWriteBuffer(conn, data)
-	_, err := conn.Write(data)
-	conn.writeMtx.Unlock()
-	return err
 }
 
 func (conn *underlayTCP) WriteMessage(mainType uint16, subType uint16, packet interface{}) error {
-	// TODO: call newMessage
+	return DefaultWriteMessage(conn, mainType, subType, packet)
-	data, err := json.Marshal(packet)
-	if err != nil {
-		return err
-	}
-	writeBytes := append(encodeHeader(mainType, subType, uint32(len(data))), data...)
-	conn.writeMtx.Lock()
-	_, err = conn.Write(writeBytes)
-	conn.writeMtx.Unlock()
-	return err
 }
 
 func (conn *underlayTCP) Close() error {
 	return conn.Conn.Close()
 }
+func (conn *underlayTCP) WLock() {
+	conn.writeMtx.Lock()
+}
+func (conn *underlayTCP) WUnlock() {
+	conn.writeMtx.Unlock()
+}
 
-func listenTCP(host string, port int, localPort int, mode string) (*underlayTCP, error) {
+func listenTCP(host string, port int, localPort int, mode string, t *P2PTunnel) (*underlayTCP, error) {
 	if mode == LinkModeTCPPunch {
-		c, err := reuse.DialTimeout("tcp", fmt.Sprintf("0.0.0.0:%d", localPort), fmt.Sprintf("%s:%d", host, port), SymmetricHandshakeAckTimeout) // TODO: timeout
+		if compareVersion(t.config.peerVersion, SyncServerTimeVersion) < 0 {
+			gLog.Printf(LvDEBUG, "peer version %s less than %s", t.config.peerVersion, SyncServerTimeVersion)
+		} else {
+			ts := time.Duration(int64(t.punchTs) + t.pn.dt - time.Now().UnixNano())
+			gLog.Printf(LvDEBUG, "sleep %d ms", ts/time.Millisecond)
+			time.Sleep(ts)
+		}
+		gLog.Println(LvDEBUG, " send tcp punch: ", fmt.Sprintf("0.0.0.0:%d", localPort), "-->", fmt.Sprintf("%s:%d", host, port))
+		c, err := reuse.DialTimeout("tcp", fmt.Sprintf("0.0.0.0:%d", localPort), fmt.Sprintf("%s:%d", host, port), CheckActiveTimeout)
 		if err != nil {
 			gLog.Println(LvDEBUG, "send tcp punch: ", err)
 			return nil, err
 		}
-		return &underlayTCP{writeMtx: &sync.Mutex{}, Conn: c}, nil
+		utcp := &underlayTCP{writeMtx: &sync.Mutex{}, Conn: c}
+		_, buff, err := utcp.ReadBuffer()
+		if err != nil {
+			return nil, fmt.Errorf("read start msg error:%s", err)
+		}
+		if buff != nil {
+			gLog.Println(LvDEBUG, string(buff))
+		}
+		utcp.WriteBytes(MsgP2P, MsgTunnelHandshakeAck, buff)
+		return utcp, nil
 	}
-	addr, _ := net.ResolveTCPAddr("tcp", fmt.Sprintf("0.0.0.0:%d", localPort))
+	t.pn.push(t.config.PeerNode, MsgPushUnderlayConnect, nil)
-	l, err := net.ListenTCP("tcp", addr)
+	tid := t.id
-	if err != nil {
+	if compareVersion(t.config.peerVersion, PublicIPVersion) < 0 { // old version
-		return nil, err
+		ipBytes := net.ParseIP(t.config.peerIP).To4()
+		tid = uint64(binary.BigEndian.Uint32(ipBytes))
+		gLog.Println(LvDEBUG, "compatible with old client, use ip as key:", tid)
 	}
-	l.SetDeadline(time.Now().Add(SymmetricHandshakeAckTimeout))
+	var utcp *underlayTCP
-	c, err := l.Accept()
+	if mode == LinkModeIntranet && gConf.Network.hasIPv4 == 0 && gConf.Network.hasUPNPorNATPMP == 0 {
-	defer l.Close()
+		addr, _ := net.ResolveTCPAddr("tcp4", fmt.Sprintf("0.0.0.0:%d", localPort))
-	if err != nil {
+		l, err := net.ListenTCP("tcp4", addr)
-		return nil, err
+		if err != nil {
+			gLog.Printf(LvERROR, "listen %d error:", localPort, err)
+			return nil, err
+		}
+		defer l.Close()
+		err = l.SetDeadline(time.Now().Add(UnderlayTCPConnectTimeout))
+		if err != nil {
+			gLog.Printf(LvERROR, "set listen timeout:", err)
+			return nil, err
+		}
+		c, err := l.Accept()
+		if err != nil {
+			return nil, err
+		}
+		utcp = &underlayTCP{writeMtx: &sync.Mutex{}, Conn: c}
+	} else {
+		if v4l != nil {
+			utcp = v4l.getUnderlayTCP(tid)
+		}
 	}
-	return &underlayTCP{writeMtx: &sync.Mutex{}, Conn: c}, nil
+
+	if utcp == nil {
+		return nil, ErrConnectPublicV4
+	}
+	return utcp, nil
 }
 
 func dialTCP(host string, port int, localPort int, mode string) (*underlayTCP, error) {
 	var c net.Conn
 	var err error
 	if mode == LinkModeTCPPunch {
-		c, err = reuse.DialTimeout("tcp", fmt.Sprintf("0.0.0.0:%d", localPort), fmt.Sprintf("%s:%d", host, port), SymmetricHandshakeAckTimeout)
+		gLog.Println(LvDev, " send tcp punch: ", fmt.Sprintf("0.0.0.0:%d", localPort), "-->", fmt.Sprintf("%s:%d", host, port))
+		if c, err = reuse.DialTimeout("tcp", fmt.Sprintf("0.0.0.0:%d", localPort), fmt.Sprintf("%s:%d", host, port), CheckActiveTimeout); err != nil {
+			gLog.Println(LvDev, "send tcp punch: ", err)
+		}
+
 	} else {
-		c, err = net.DialTimeout("tcp", fmt.Sprintf("%s:%d", host, port), SymmetricHandshakeAckTimeout)
+		c, err = net.DialTimeout("tcp", fmt.Sprintf("%s:%d", host, port), CheckActiveTimeout)
 	}
 
 	if err != nil {
-		gLog.Printf(LvERROR, "Dial %s:%d error:%s", host, port, err)
+		gLog.Printf(LvDev, "Dial %s:%d error:%s", host, port, err)
 		return nil, err
 	}
+	tc := c.(*net.TCPConn)
+	tc.SetKeepAlive(true)
+	tc.SetKeepAlivePeriod(UnderlayTCPKeepalive)
 	gLog.Printf(LvDEBUG, "Dial %s:%d OK", host, port)
 	return &underlayTCP{writeMtx: &sync.Mutex{}, Conn: c}, nil
 }

core/underlay_tcp6.go

@@ -1,16 +1,13 @@
 package openp2p
 
 import (
-	"encoding/json"
 	"fmt"
-	"io"
 	"net"
 	"sync"
 	"time"
 )
 
 type underlayTCP6 struct {
-	listener net.Listener
 	writeMtx *sync.Mutex
 	net.Conn
 }
@@ -20,60 +17,38 @@ func (conn *underlayTCP6) Protocol() string {
 }
 
 func (conn *underlayTCP6) ReadBuffer() (*openP2PHeader, []byte, error) {
-	headBuf := make([]byte, openP2PHeaderSize)
+	return DefaultReadBuffer(conn)
-	_, err := io.ReadFull(conn, headBuf)
-	if err != nil {
-		return nil, nil, err
-	}
-	head, err := decodeHeader(headBuf)
-	if err != nil {
-		return nil, nil, err
-	}
-	dataBuf := make([]byte, head.DataLen)
-	_, err = io.ReadFull(conn, dataBuf)
-	return head, dataBuf, err
 }
 
 func (conn *underlayTCP6) WriteBytes(mainType uint16, subType uint16, data []byte) error {
-	writeBytes := append(encodeHeader(mainType, subType, uint32(len(data))), data...)
+	return DefaultWriteBytes(conn, mainType, subType, data)
-	conn.writeMtx.Lock()
-	_, err := conn.Write(writeBytes)
-	conn.writeMtx.Unlock()
-	return err
 }
 
 func (conn *underlayTCP6) WriteBuffer(data []byte) error {
-	conn.writeMtx.Lock()
+	return DefaultWriteBuffer(conn, data)
-	_, err := conn.Write(data)
-	conn.writeMtx.Unlock()
-	return err
 }
 
 func (conn *underlayTCP6) WriteMessage(mainType uint16, subType uint16, packet interface{}) error {
-	// TODO: call newMessage
+	return DefaultWriteMessage(conn, mainType, subType, packet)
-	data, err := json.Marshal(packet)
-	if err != nil {
-		return err
-	}
-	writeBytes := append(encodeHeader(mainType, subType, uint32(len(data))), data...)
-	conn.writeMtx.Lock()
-	_, err = conn.Write(writeBytes)
-	conn.writeMtx.Unlock()
-	return err
 }
 
 func (conn *underlayTCP6) Close() error {
 	return conn.Conn.Close()
 }
-
+func (conn *underlayTCP6) WLock() {
-func listenTCP6(port int, idleTimeout time.Duration) (*underlayTCP6, error) {
+	conn.writeMtx.Lock()
+}
+func (conn *underlayTCP6) WUnlock() {
+	conn.writeMtx.Unlock()
+}
+func listenTCP6(port int, timeout time.Duration) (*underlayTCP6, error) {
 	addr, _ := net.ResolveTCPAddr("tcp6", fmt.Sprintf("[::]:%d", port))
 	l, err := net.ListenTCP("tcp6", addr)
 	if err != nil {
 		return nil, err
 	}
 	defer l.Close()
-	l.SetDeadline(time.Now().Add(SymmetricHandshakeAckTimeout))
+	l.SetDeadline(time.Now().Add(timeout))
 	c, err := l.Accept()
 	defer l.Close()
 	if err != nil {
@@ -83,7 +58,7 @@ func listenTCP6(port int, idleTimeout time.Duration) (*underlayTCP6, error) {
 }
 
 func dialTCP6(host string, port int) (*underlayTCP6, error) {
-	c, err := net.DialTimeout("tcp6", fmt.Sprintf("[%s]:%d", host, port), SymmetricHandshakeAckTimeout)
+	c, err := net.DialTimeout("tcp6", fmt.Sprintf("[%s]:%d", host, port), UnderlayConnectTimeout)
 	if err != nil {
 		gLog.Printf(LvERROR, "Dial %s:%d error:%s", host, port, err)
 		return nil, err

core/update.go

@@ -1,193 +1,239 @@
-package openp2p
+package openp2p
-
+
-import (
+import (
-	"archive/tar"
+	"archive/tar"
-	"archive/zip"
+	"archive/zip"
-	"compress/gzip"
+	"compress/gzip"
-	"crypto/tls"
+	"crypto/tls"
-	"encoding/json"
+	"crypto/x509"
-	"fmt"
+	"encoding/json"
-	"io"
+	"fmt"
-	"io/ioutil"
+	"io"
-	"net/http"
+	"io/ioutil"
-	"os"
+	"net/http"
-	"path/filepath"
+	"net/url"
-	"runtime"
+	"os"
-	"time"
+	"path/filepath"
-)
+	"runtime"
-
+	"time"
-func update(host string, port int) {
+)
-	gLog.Println(LvINFO, "update start")
+
-	defer gLog.Println(LvINFO, "update end")
+func update(host string, port int) error {
-	c := http.Client{
+	gLog.Println(LvINFO, "update start")
-		Transport: &http.Transport{
+	defer gLog.Println(LvINFO, "update end")
-			TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
+	caCertPool, err := x509.SystemCertPool()
-		},
+	if err != nil {
-		Timeout: time.Second * 30,
+		gLog.Println(LvERROR, "Failed to load system root CAs:", err)
-	}
+	} else {
-	goos := runtime.GOOS
+		caCertPool = x509.NewCertPool()
-	goarch := runtime.GOARCH
+	}
-	rsp, err := c.Get(fmt.Sprintf("https://%s:%d/api/v1/update?fromver=%s&os=%s&arch=%s", host, port, OpenP2PVersion, goos, goarch))
+	caCertPool.AppendCertsFromPEM([]byte(rootCA))
-	if err != nil {
+	caCertPool.AppendCertsFromPEM([]byte(ISRGRootX1))
-		gLog.Println(LvERROR, "update:query update list failed:", err)
+
-		return
+	c := http.Client{
-	}
+		Transport: &http.Transport{
-	defer rsp.Body.Close()
+			TLSClientConfig: &tls.Config{RootCAs: caCertPool,
-	if rsp.StatusCode != http.StatusOK {
+				InsecureSkipVerify: false},
-		gLog.Println(LvERROR, "get update info error:", rsp.Status)
+		},
-		return
+		Timeout: time.Second * 30,
-	}
+	}
-	rspBuf, err := ioutil.ReadAll(rsp.Body)
+	goos := runtime.GOOS
-	if err != nil {
+	goarch := runtime.GOARCH
-		gLog.Println(LvERROR, "update:read update list failed:", err)
+	rsp, err := c.Get(fmt.Sprintf("https://%s:%d/api/v1/update?fromver=%s&os=%s&arch=%s&user=%s&node=%s", host, port, OpenP2PVersion, goos, goarch, url.QueryEscape(gConf.Network.User), url.QueryEscape(gConf.Network.Node)))
-		return
+	if err != nil {
-	}
+		gLog.Println(LvERROR, "update:query update list failed:", err)
-	updateInfo := UpdateInfo{}
+		return err
-	err = json.Unmarshal(rspBuf, &updateInfo)
+	}
-	if err != nil {
+	defer rsp.Body.Close()
-		gLog.Println(LvERROR, rspBuf, " update info decode error:", err)
+	if rsp.StatusCode != http.StatusOK {
-		return
+		gLog.Println(LvERROR, "get update info error:", rsp.Status)
-	}
+		return err
-	if updateInfo.Error != 0 {
+	}
-		gLog.Println(LvERROR, "update error:", updateInfo.Error, updateInfo.ErrorDetail)
+	rspBuf, err := ioutil.ReadAll(rsp.Body)
-		return
+	if err != nil {
-	}
+		gLog.Println(LvERROR, "update:read update list failed:", err)
-	err = updateFile(updateInfo.Url, "", "openp2p")
+		return err
-	if err != nil {
+	}
-		gLog.Println(LvERROR, "update: download failed:", err)
+	updateInfo := UpdateInfo{}
-		return
+	if err = json.Unmarshal(rspBuf, &updateInfo); err != nil {
-	}
+		gLog.Println(LvERROR, rspBuf, " update info decode error:", err)
-}
+		return err
-
+	}
-// todo rollback on error
+	if updateInfo.Error != 0 {
-func updateFile(url string, checksum string, dst string) error {
+		gLog.Println(LvERROR, "update error:", updateInfo.Error, updateInfo.ErrorDetail)
-	gLog.Println(LvINFO, "download ", url)
+		return err
-	tmpFile := filepath.Dir(os.Args[0]) + "/openp2p.tmp"
+	}
-	output, err := os.OpenFile(tmpFile, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0776)
+	err = updateFile(updateInfo.Url, "", "openp2p")
-	if err != nil {
+	if err != nil {
-		gLog.Printf(LvERROR, "OpenFile %s error:%s", tmpFile, err)
+		gLog.Println(LvERROR, "update: download failed:", err)
-		return err
+		return err
-	}
+	}
-	tr := &http.Transport{
+	return nil
-		TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
+}
-	}
+
-	client := &http.Client{Transport: tr}
+func downloadFile(url string, checksum string, dstFile string) error {
-	response, err := client.Get(url)
+	output, err := os.OpenFile(dstFile, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0776)
-	if err != nil {
+	if err != nil {
-		gLog.Printf(LvERROR, "download url %s error:%s", url, err)
+		gLog.Printf(LvERROR, "OpenFile %s error:%s", dstFile, err)
-		output.Close()
+		return err
-		return err
+	}
-	}
+	caCertPool, err := x509.SystemCertPool()
-	defer response.Body.Close()
+	if err != nil {
-	n, err := io.Copy(output, response.Body)
+		gLog.Println(LvERROR, "Failed to load system root CAs:", err)
-	if err != nil {
+	} else {
-		gLog.Printf(LvERROR, "io.Copy error:%s", err)
+		caCertPool = x509.NewCertPool()
-		output.Close()
+	}
-		return err
+	caCertPool.AppendCertsFromPEM([]byte(rootCA))
-	}
+	caCertPool.AppendCertsFromPEM([]byte(ISRGRootX1))
-	output.Sync()
+	tr := &http.Transport{
-	output.Close()
+		TLSClientConfig: &tls.Config{
-	gLog.Println(LvINFO, "download ", url, " ok")
+			RootCAs:            caCertPool,
-	gLog.Printf(LvINFO, "size: %d bytes", n)
+			InsecureSkipVerify: false},
-
+	}
-	err = os.Rename(os.Args[0], os.Args[0]+"0")
+	client := &http.Client{Transport: tr}
-	if err != nil && os.IsExist(err) {
+	response, err := client.Get(url)
-		gLog.Printf(LvINFO, " rename %s error:%s", os.Args[0], err)
+	if err != nil {
-	}
+		gLog.Printf(LvERROR, "download url %s error:%s", url, err)
-	// extract
+		output.Close()
-	gLog.Println(LvINFO, "extract files")
+		return err
-	err = extract(filepath.Dir(os.Args[0]), tmpFile)
+	}
-	if err != nil {
+	defer response.Body.Close()
-		gLog.Printf(LvERROR, "extract error:%s. revert rename", err)
+	n, err := io.Copy(output, response.Body)
-		os.Rename(os.Args[0]+"0", os.Args[0])
+	if err != nil {
-		return err
+		gLog.Printf(LvERROR, "io.Copy error:%s", err)
-	}
+		output.Close()
-	os.Remove(tmpFile)
+		return err
-	return nil
+	}
-}
+	output.Sync()
-
+	output.Close()
-func extract(dst, src string) (err error) {
+	gLog.Println(LvINFO, "download ", url, " ok")
-	if runtime.GOOS == "windows" {
+	gLog.Printf(LvINFO, "size: %d bytes", n)
-		return unzip(dst, src)
+	return nil
-	} else {
+}
-		return extractTgz(dst, src)
+
-	}
+func updateFile(url string, checksum string, dst string) error {
-}
+	gLog.Println(LvINFO, "download ", url)
-
+	tmpFile := filepath.Dir(os.Args[0]) + "/openp2p.tmp"
-func unzip(dst, src string) (err error) {
+	err := downloadFile(url, checksum, tmpFile)
-	archive, err := zip.OpenReader(src)
+	if err != nil {
-	if err != nil {
+		return err
-		return err
+	}
-	}
+	backupFile := os.Args[0] + "0"
-	defer archive.Close()
+	err = os.Rename(os.Args[0], backupFile) // the old daemon process was using the 0 file, so it will prevent override it
-
+	if err != nil {
-	for _, f := range archive.File {
+		gLog.Printf(LvINFO, " rename %s error:%s, retry 1", os.Args[0], err)
-		filePath := filepath.Join(dst, f.Name)
+		backupFile = os.Args[0] + "1"
-		fmt.Println("unzipping file ", filePath)
+		err = os.Rename(os.Args[0], backupFile)
-		if f.FileInfo().IsDir() {
+		if err != nil {
-			fmt.Println("creating directory...")
+			gLog.Printf(LvINFO, " rename %s error:%s", os.Args[0], err)
-			os.MkdirAll(filePath, os.ModePerm)
+		}
-			continue
+	}
-		}
+	// extract
-		if err := os.MkdirAll(filepath.Dir(filePath), os.ModePerm); err != nil {
+	gLog.Println(LvINFO, "extract files")
-			return err
+	err = extract(filepath.Dir(os.Args[0]), tmpFile)
-		}
+	if err != nil {
-		dstFile, err := os.OpenFile(filePath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, f.Mode())
+		gLog.Printf(LvERROR, "extract error:%s. revert rename", err)
-		if err != nil {
+		os.Rename(backupFile, os.Args[0])
-			return err
+		return err
-		}
+	}
-		fileInArchive, err := f.Open()
+	os.Remove(tmpFile)
-		if err != nil {
+	return nil
-			return err
+}
-		}
+
-		if _, err := io.Copy(dstFile, fileInArchive); err != nil {
+func extract(dst, src string) (err error) {
-			return err
+	if runtime.GOOS == "windows" {
-		}
+		return unzip(dst, src)
-		dstFile.Close()
+	} else {
-		fileInArchive.Close()
+		return extractTgz(dst, src)
-	}
+	}
-	return nil
+}
-}
+
-
+func unzip(dst, src string) (err error) {
-func extractTgz(dst, src string) error {
+	archive, err := zip.OpenReader(src)
-	gzipStream, err := os.Open(src)
+	if err != nil {
-	if err != nil {
+		return err
-		return err
+	}
-	}
+	defer archive.Close()
-	uncompressedStream, err := gzip.NewReader(gzipStream)
+
-	if err != nil {
+	for _, f := range archive.File {
-		return err
+		filePath := filepath.Join(dst, f.Name)
-	}
+		fmt.Println("unzipping file ", filePath)
-	tarReader := tar.NewReader(uncompressedStream)
+		if f.FileInfo().IsDir() {
-	for {
+			fmt.Println("creating directory...")
-		header, err := tarReader.Next()
+			os.MkdirAll(filePath, os.ModePerm)
-		if err == io.EOF {
+			continue
-			break
+		}
-		}
+		if err := os.MkdirAll(filepath.Dir(filePath), os.ModePerm); err != nil {
-		if err != nil {
+			return err
-			return err
+		}
-		}
+		dstFile, err := os.OpenFile(filePath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, f.Mode())
-		switch header.Typeflag {
+		if err != nil {
-		case tar.TypeDir:
+			return err
-			if err := os.Mkdir(header.Name, 0755); err != nil {
+		}
-				return err
+		fileInArchive, err := f.Open()
-			}
+		if err != nil {
-		case tar.TypeReg:
+			return err
-			filePath := filepath.Join(dst, header.Name)
+		}
-			outFile, err := os.OpenFile(filePath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, os.FileMode(header.Mode))
+		if _, err := io.Copy(dstFile, fileInArchive); err != nil {
-			if err != nil {
+			return err
-				return err
+		}
-			}
+		dstFile.Close()
-			if err != nil {
+		fileInArchive.Close()
-				return err
+	}
-			}
+	return nil
-			defer outFile.Close()
+}
-			if _, err := io.Copy(outFile, tarReader); err != nil {
+
-				return err
+func extractTgz(dst, src string) error {
-			}
+	gzipStream, err := os.Open(src)
-		default:
+	if err != nil {
-			return err
+		return err
-		}
+	}
-	}
+	uncompressedStream, err := gzip.NewReader(gzipStream)
-	return nil
+	if err != nil {
-}
+		return err
+	}
+	tarReader := tar.NewReader(uncompressedStream)
+	for {
+		header, err := tarReader.Next()
+		if err == io.EOF {
+			break
+		}
+		if err != nil {
+			return err
+		}
+		switch header.Typeflag {
+		case tar.TypeDir:
+			if err := os.Mkdir(header.Name, 0755); err != nil {
+				return err
+			}
+		case tar.TypeReg:
+			filePath := filepath.Join(dst, header.Name)
+			outFile, err := os.OpenFile(filePath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, os.FileMode(header.Mode))
+			if err != nil {
+				return err
+			}
+			defer outFile.Close()
+			if _, err := io.Copy(outFile, tarReader); err != nil {
+				return err
+			}
+		default:
+			return err
+		}
+	}
+	return nil
+}
+
+func cleanTempFiles() {
+	tmpFile := os.Args[0] + "0"
+	if _, err := os.Stat(tmpFile); err == nil {
+		if err := os.Remove(tmpFile); err != nil {
+			gLog.Printf(LvDEBUG, " remove %s error:%s", tmpFile, err)
+		}
+	}
+	tmpFile = os.Args[0] + "1"
+	if _, err := os.Stat(tmpFile); err == nil {
+		if err := os.Remove(tmpFile); err != nil {
+			gLog.Printf(LvDEBUG, " remove %s error:%s", tmpFile, err)
+		}
+	}
+}

core/upnp.go

@@ -5,6 +5,7 @@ package openp2p
 
 import (
 	"bytes"
+	"crypto/tls"
 	"encoding/xml"
 	"errors"
 	"fmt"
@@ -181,7 +182,12 @@ func localIPv4() string { // TODO: multi nic will wrong
 }
 
 func getServiceURL(rootURL string) (url, urnDomain string, err error) {
-	r, err := http.Get(rootURL)
+	client := &http.Client{
+		Transport: &http.Transport{
+			TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
+		},
+		Timeout: time.Second * 3}
+	r, err := client.Get(rootURL)
 	if err != nil {
 		return
 	}
@@ -352,11 +358,6 @@ func (n *upnpNAT) AddPortMapping(protocol string, externalPort, internalPort int
 		return
 	}
 
-	// TODO: check response to see if the port was forwarded
-	// log.Println(message, response)
-	// JAE:
-	// body, err := ioutil.ReadAll(response.Body)
-	// fmt.Println(string(body), err)
 	mappedExternalPort = externalPort
 	_ = response
 	return
@@ -378,8 +379,6 @@ func (n *upnpNAT) DeletePortMapping(protocol string, externalPort, internalPort
 		return
 	}
 
-	// TODO: check response to see if the port was deleted
-	// log.Println(message, response)
 	_ = response
 	return
 }

core/util_darwin.go

@@ -21,7 +21,7 @@ func setRLimit() error {
 	if err := syscall.Getrlimit(syscall.RLIMIT_NOFILE, &limit); err != nil {
 		return err
 	}
-	limit.Cur = 10240
+	limit.Cur = 65536
 	if err := syscall.Setrlimit(syscall.RLIMIT_NOFILE, &limit); err != nil {
 		return err
 	}

core/util_freebsd.go

@@ -0,0 +1,69 @@
+package openp2p
+
+import (
+	"bufio"
+	"bytes"
+	"io/ioutil"
+	"os"
+	"runtime"
+	"strings"
+	"syscall"
+)
+
+const (
+	defaultInstallPath = "/usr/local/openp2p"
+	defaultBinName     = "openp2p"
+)
+
+func getOsName() (osName string) {
+	var sysnamePath string
+	sysnamePath = "/etc/redhat-release"
+	_, err := os.Stat(sysnamePath)
+	if err != nil && os.IsNotExist(err) {
+		str := "PRETTY_NAME="
+		f, err := os.Open("/etc/os-release")
+		if err == nil {
+			buf := bufio.NewReader(f)
+			for {
+				line, err := buf.ReadString('\n')
+				if err == nil {
+					line = strings.TrimSpace(line)
+					pos := strings.Count(line, str)
+					if pos > 0 {
+						len1 := len([]rune(str)) + 1
+						rs := []rune(line)
+						osName = string(rs[len1 : (len(rs))-1])
+						break
+					}
+				} else {
+					break
+				}
+			}
+		}
+	} else {
+		buff, err := ioutil.ReadFile(sysnamePath)
+		if err == nil {
+			osName = string(bytes.TrimSpace(buff))
+		}
+	}
+	if osName == "" {
+		osName = "FreeBSD"
+	}
+	return
+}
+
+func setRLimit() error {
+	var limit syscall.Rlimit
+	if err := syscall.Getrlimit(syscall.RLIMIT_NOFILE, &limit); err != nil {
+		return err
+	}
+	limit.Max = 65536
+	limit.Cur = limit.Max
+	if err := syscall.Setrlimit(syscall.RLIMIT_NOFILE, &limit); err != nil {
+		return err
+	}
+	return nil
+}
+
+func setFirewall() {
+}

core/util_linux.go

@@ -64,7 +64,7 @@ func setRLimit() error {
 	if err := syscall.Getrlimit(syscall.RLIMIT_NOFILE, &limit); err != nil {
 		return err
 	}
-	limit.Max = 1024 * 1024
+	limit.Max = 65536
 	limit.Cur = limit.Max
 	if err := syscall.Setrlimit(syscall.RLIMIT_NOFILE, &limit); err != nil {
 		return err

core/util_windows.go

@@ -5,6 +5,7 @@ import (
 	"os"
 	"os/exec"
 	"path/filepath"
+	"strconv"
 	"strings"
 
 	"golang.org/x/sys/windows/registry"
@@ -23,6 +24,17 @@ func getOsName() (osName string) {
 	defer k.Close()
 	pn, _, err := k.GetStringValue("ProductName")
 	if err == nil {
+		currentBuild, _, err := k.GetStringValue("CurrentBuild")
+		if err != nil {
+			return
+		}
+		buildNumber, err := strconv.Atoi(currentBuild)
+		if err != nil {
+			return
+		}
+		if buildNumber >= 22000 {
+			pn = strings.Replace(pn, "Windows 10", "Windows 11", 1)
+		}
 		osName = pn
 	}
 	return

core/v4listener.go

@@ -0,0 +1,92 @@
+package openp2p
+
+import (
+	"encoding/binary"
+	"fmt"
+	"net"
+	"sync"
+	"time"
+)
+
+type v4Listener struct {
+	conns    sync.Map
+	port     int
+	acceptCh chan bool
+}
+
+func (vl *v4Listener) start() error {
+	v4l.acceptCh = make(chan bool, 500)
+	for {
+		vl.listen()
+		time.Sleep(UnderlayTCPConnectTimeout)
+	}
+}
+
+func (vl *v4Listener) listen() error {
+	gLog.Printf(LvINFO, "v4Listener listen %d start", vl.port)
+	defer gLog.Printf(LvINFO, "v4Listener listen %d end", vl.port)
+	addr, _ := net.ResolveTCPAddr("tcp4", fmt.Sprintf("0.0.0.0:%d", vl.port))
+	l, err := net.ListenTCP("tcp4", addr)
+	if err != nil {
+		gLog.Printf(LvERROR, "v4Listener listen %d error:", vl.port, err)
+		return err
+	}
+	defer l.Close()
+	for {
+		c, err := l.Accept()
+		if err != nil {
+			break
+		}
+		go vl.handleConnection(c)
+	}
+	return nil
+}
+func (vl *v4Listener) handleConnection(c net.Conn) {
+	gLog.Println(LvDEBUG, "v4Listener accept connection: ", c.RemoteAddr().String())
+	utcp := &underlayTCP{writeMtx: &sync.Mutex{}, Conn: c, connectTime: time.Now()}
+	utcp.SetReadDeadline(time.Now().Add(UnderlayTCPConnectTimeout))
+	_, buff, err := utcp.ReadBuffer()
+	if err != nil {
+		gLog.Printf(LvERROR, "utcp.ReadBuffer error:", err)
+	}
+	utcp.WriteBytes(MsgP2P, MsgTunnelHandshakeAck, buff)
+	var tid uint64
+	if string(buff) == "OpenP2P,hello" { // old client
+		// save remoteIP as key
+		remoteAddr := c.RemoteAddr().(*net.TCPAddr).IP
+		ipBytes := remoteAddr.To4()
+		tid = uint64(binary.BigEndian.Uint32(ipBytes)) // bytes not enough for uint64
+		gLog.Println(LvDEBUG, "hello ", string(buff))
+	} else {
+		if len(buff) < 8 {
+			return
+		}
+		tid = binary.LittleEndian.Uint64(buff[:8])
+		gLog.Println(LvDEBUG, "hello ", tid)
+	}
+	// clear timeout connection
+	vl.conns.Range(func(idx, i interface{}) bool {
+		ut := i.(*underlayTCP)
+		if ut.connectTime.Before(time.Now().Add(-UnderlayTCPConnectTimeout)) {
+			vl.conns.Delete(idx)
+		}
+		return true
+	})
+	vl.conns.Store(tid, utcp)
+	if len(vl.acceptCh) == 0 {
+		vl.acceptCh <- true
+	}
+}
+
+func (vl *v4Listener) getUnderlayTCP(tid uint64) *underlayTCP {
+	for i := 0; i < 100; i++ {
+		select {
+		case <-time.After(time.Millisecond * 50):
+		case <-vl.acceptCh:
+		}
+		if u, ok := vl.conns.LoadAndDelete(tid); ok {
+			return u.(*underlayTCP)
+		}
+	}
+	return nil
+}

docker/Dockerfile

@@ -0,0 +1,12 @@
+FROM  alpine:3.18.2
+
+# Replace the default Alpine repositories with Aliyun mirrors
+RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories && \
+    apk add --no-cache ca-certificates iptables && \
+    rm -rf /tmp/* /var/tmp/* /var/cache/apk/* /var/cache/distfiles/*
+
+COPY get-client.sh /
+ARG DOCKER_VER="latest"
+RUN echo $TARGETPLATFORM && chmod +x /get-client.sh && ./get-client.sh
+
+ENTRYPOINT ["/openp2p"]

docker/get-client.sh

@@ -0,0 +1,45 @@
+#!/bin/sh
+
+
+echo "Building version:${DOCKER_VER}"
+echo "Running on platform: $TARGETPLATFORM"
+# TARGETPLATFORM=$(echo $TARGETPLATFORM | tr ',' '/')
+echo "Running on platform: $TARGETPLATFORM"
+sysType="linux-amd64"
+		archType=$(uname -m)
+		if [[ $archType == aarch64 ]] ;
+		then
+		    sysType="linux-arm64"
+		elif  [[ $archType == arm* ]] ;
+		then
+			sysType="linux-arm"
+		elif  [[ $archType == i*86 ]] ;
+		then
+			sysType="linux-386"
+		elif  [[ $archType == mips ]] ;
+		then
+			sysType="linux-mipsle"
+			ls /lib |grep mipsel
+			if [[ $? -ne 0 ]]; then
+				# mipsel not found, it's mipseb
+				sysType="linux-mipsbe"
+			fi
+		fi
+url="https://openp2p.cn/download/v1/${DOCKER_VER}/openp2p-latest.$sysType.tar.gz"
+echo "download $url start"
+
+if [ -f /usr/bin/curl ]; then
+	curl -k -o openp2p.tar.gz $url
+else
+  wget --no-check-certificate -O openp2p.tar.gz $url
+fi
+if [ $? -ne 0 ]; then
+	echo "download error $?"
+    exit 9
+fi
+echo "download ok"
+tar -xzvf openp2p.tar.gz
+chmod +x openp2p
+pwd
+ls -l
+exit 0

example/dll/dll.cpp

@@ -0,0 +1,14 @@
+#include <iostream>
+#include <windows.h>
+
+using namespace std;
+typedef void (*pRun)(const char *);
+
+int main(int argc, char *argv[])
+{
+	HMODULE dll = LoadLibraryA("openp2p.dll");
+	pRun run = (pRun)GetProcAddress(dll, "RunCmd");
+	run("-node 5800-debug2 -token YOUR-TOKEN");
+	FreeLibrary(dll);
+	return 0;
+}

example/echo/echo-client.go

@@ -0,0 +1,42 @@
+package main
+
+import (
+	"fmt"
+	op "openp2p/core"
+	"time"
+)
+
+func main() {
+	op.Run()
+	for i := 0; i < 10; i++ {
+		go echoClient("5800-debug")
+	}
+	echoClient("5800-debug")
+}
+
+func echoClient(peerNode string) {
+	sendDatalen := op.ReadBuffLen
+	sendBuff := make([]byte, sendDatalen)
+	for i := 0; i < len(sendBuff); i++ {
+		sendBuff[i] = byte('A' + i/100)
+	}
+	// peerNode = "YOUR-PEER-NODE-NAME"
+	if err := op.GNetwork.ConnectNode(peerNode); err != nil {
+		fmt.Println("connect error:", err)
+		return
+	}
+	for i := 0; ; i++ {
+		sendBuff[1] = 'A' + byte(i%26)
+		if err := op.GNetwork.WriteNode(op.NodeNameToID(peerNode), sendBuff[:sendDatalen]); err != nil {
+			fmt.Println("write error:", err)
+			break
+		}
+		nd := op.GNetwork.ReadNode(time.Second * 10)
+		if nd == nil {
+			fmt.Printf("waiting for node data\n")
+			time.Sleep(time.Second * 10)
+			continue
+		}
+		fmt.Printf("read %d len=%d data=%s\n", nd.NodeID, len(nd.Data), nd.Data[:16]) // only print 16 bytes
+	}
+}

example/echo/echo-server.go

@@ -0,0 +1,32 @@
+package main
+
+import (
+	"fmt"
+	op "openp2p/core"
+	"time"
+)
+
+func main() {
+	op.Run()
+	echoServer()
+	forever := make(chan bool)
+	<-forever
+}
+
+func echoServer() {
+	// peerID := fmt.Sprintf("%d", core.NodeNameToID(peerNode))
+	for {
+		nd := op.GNetwork.ReadNode(time.Second * 10)
+		if nd == nil {
+			fmt.Printf("waiting for node data\n")
+			// time.Sleep(time.Second * 10)
+			continue
+		}
+		// fmt.Printf("read %s len=%d data=%s\n", nd.Node, len(nd.Data), nd.Data[:16])
+		nd.Data[0] = 'R' // echo server mark as replied
+		if err := op.GNetwork.WriteNode(nd.NodeID, nd.Data); err != nil {
+			fmt.Println("write error:", err)
+			break
+		}
+	}
+}

go.mod

@@ -1,28 +1,42 @@
 module openp2p
 
-go 1.18
+go 1.20
 
 require (
+	github.com/emirpasic/gods v1.18.1
 	github.com/gorilla/websocket v1.4.2
-	github.com/lucas-clemente/quic-go v0.27.0
 	github.com/openp2p-cn/go-reuseport v0.3.2
 	github.com/openp2p-cn/service v1.0.0
-	golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f
+	github.com/openp2p-cn/totp v0.0.0-20230421034602-0f3320ffb25e
+	github.com/openp2p-cn/wireguard-go v0.0.20240223
+	github.com/quic-go/quic-go v0.34.0
+	github.com/vishvananda/netlink v1.1.0
+	github.com/xtaci/kcp-go/v5 v5.5.17
+	golang.org/x/sys v0.21.0
+	golang.zx2c4.com/wireguard/windows v0.5.3
 )
 
 require (
-	github.com/cheekybits/genny v1.0.0 // indirect
-	github.com/fsnotify/fsnotify v1.4.9 // indirect
 	github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0 // indirect
+	github.com/golang/mock v1.6.0 // indirect
+	github.com/google/pprof v0.0.0-20210407192527-94a9f03dee38 // indirect
 	github.com/kardianos/service v1.2.2 // indirect
-	github.com/marten-seemann/qtls-go1-16 v0.1.5 // indirect
+	github.com/klauspost/cpuid/v2 v2.2.5 // indirect
-	github.com/marten-seemann/qtls-go1-17 v0.1.1 // indirect
+	github.com/klauspost/reedsolomon v1.11.8 // indirect
-	github.com/marten-seemann/qtls-go1-18 v0.1.1 // indirect
+	github.com/onsi/ginkgo/v2 v2.2.0 // indirect
-	github.com/nxadm/tail v1.4.8 // indirect
+	github.com/pkg/errors v0.9.1 // indirect
-	github.com/onsi/ginkgo v1.16.4 // indirect
+	github.com/quic-go/qtls-go1-19 v0.3.2 // indirect
-	golang.org/x/crypto v0.0.0-20210921155107-089bfa567519 // indirect
+	github.com/quic-go/qtls-go1-20 v0.2.2 // indirect
-	golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4 // indirect
+	github.com/templexxx/cpu v0.1.0 // indirect
-	golang.org/x/net v0.0.0-20220722155237-a158d28d115b // indirect
+	github.com/templexxx/xorsimd v0.4.2 // indirect
-	golang.org/x/tools v0.1.12 // indirect
+	github.com/tjfoc/gmsm v1.4.1 // indirect
-	gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 // indirect
+	github.com/vishvananda/netns v0.0.0-20191106174202-0a2b9b5464df // indirect
+	golang.org/x/crypto v0.24.0 // indirect
+	golang.org/x/exp v0.0.0-20221205204356-47842c84f3db // indirect
+	golang.org/x/mod v0.18.0 // indirect
+	golang.org/x/net v0.26.0 // indirect
+	golang.org/x/tools v0.22.0 // indirect
+	golang.zx2c4.com/wintun v0.0.0-20230126152724-0fa3db229ce2 // indirect
+	golang.zx2c4.com/wireguard v0.0.0-20231211153847-12269c276173 // indirect
+	google.golang.org/protobuf v1.28.1 // indirect
 )

lib/openp2p.go

@@ -0,0 +1,18 @@
+package main
+
+// On Windows env
+// cd lib
+// go build -o openp2p.dll -buildmode=c-shared openp2p.go
+// caller example see example/dll
+import (
+	op "openp2p/core"
+)
+import "C"
+
+func main() {
+}
+
+//export RunCmd
+func RunCmd(cmd *C.char) {
+	op.RunCmd(C.GoString(cmd))
+}