refactor: harden sqlite runtime and packaged startup flow

- refine config loading with --config and current-directory .env

- improve sqlite busy handling and runtime DB tuning

- restore telegram payment notification and wallet input flow

- add behavior-based config and callback recovery tests
This commit is contained in:
alphago9
2026-03-31 13:54:10 +08:00
parent 08f0c93e11
commit 682aa3907a
23 changed files with 975 additions and 133 deletions
+3
View File
@@ -1,5 +1,8 @@
.idea/
src/runtime/
src/.env
src/*.db
src/*.db-shm
src/*.db-wal
.DS_Store
.env
+157
View File
@@ -0,0 +1,157 @@
# Development Standard
## Purpose
This document defines the default development goals, review criteria, and testing constraints for changes in this repository.
Before starting any implementation, use this document as a checklist.
## Development Goals
Every change should be judged against all of the following goals:
1. Functional completeness
The target feature must actually work end to end, not just compile.
2. No accidental teammate regression
Do not accidentally remove or override teammate capabilities that already exist on the base branch.
3. Robustness
The code should handle normal failure cases, retries, state conflicts, and restart scenarios as reasonably as possible.
4. Elegance
Prefer clear responsibilities, minimal coupling, and a small number of well-named moving parts.
5. Maintainability
Avoid hidden behavior, confusing configuration, or implementation details that make future debugging harder.
6. No dirty output
Never commit local runtime artifacts, local databases, caches, logs, or other development byproducts.
7. No mojibake
Source files, messages, and docs should not contain broken encoding text.
## Multi-Dimensional Self Review
Each meaningful change should be evaluated across these dimensions:
- functional completeness
- engineering implementation quality
- robustness
- elegance
- maintainability
- regression risk
- commit readiness
- test coverage quality
Scoring should be honest. High scores require both working behavior and clean delivery quality.
## Code Quality Rules
### Redundancy
Avoid redundant code unless duplication is clearly cheaper and safer than abstraction.
The following are considered bad redundancy:
- duplicate business logic with only tiny naming differences
- duplicate retry or state logic in multiple places
- helper functions that exist only to wrap one line without improving clarity
The following are acceptable:
- small data shapes for read optimization
- isolated helpers that remove repeated infrastructure setup
- compatibility shims with clear removal intent
### Elegance
Code is considered elegant when:
- responsibilities are split cleanly
- names explain intent
- state transitions are explicit
- configuration is understandable
- behavior is observable through useful logs
- failure handling does not hide important problems
### Robustness
Code is considered robust when:
- state changes are guarded by conditions where needed
- retries are controlled, not infinite
- restart recovery is possible where it matters
- local concurrency does not easily corrupt behavior
- failure in one side task does not silently break the main flow
## Bug Standard
Ask these questions before treating a change as finished:
- Is there any blocking functional bug?
- Is there any likely edge-case bug?
- Is there any operational bug under realistic local concurrency?
- Is there any user-visible regression?
If a bug is known, it must be called out explicitly in review notes.
## Testing Rules
### Required Principle
Tests must validate behavior, not re-implement the code under test.
### Forbidden Test Types
Do not write:
- invalid tests
- mirror tests
- tests that only repeat implementation branches line by line
- tests that only verify mocks interacted in the same order as the code was written
- tests that lock onto incidental internals instead of real behavior
### Preferred Test Types
Prefer:
- end-to-end behavior tests
- state transition tests
- concurrency conflict tests
- retry and recovery tests
- expiration and timing behavior tests
- persistence and restart-oriented tests where relevant
### Test Coverage Expectations
Coverage quality matters more than raw coverage count.
For core flows, tests should cover:
- normal success path
- important failure path
- state conflict or idempotency path
- retry or delayed processing path where relevant
## Review Checklist
Before considering a change ready, confirm:
- functionality is complete
- teammate capability was not accidentally lost
- no local artifact will be committed
- no mojibake exists
- no obvious redundant code exists
- no known blocking bug remains
- robustness is acceptable for the target deployment model
- tests are behavior-based and not mirror tests
- review notes clearly explain intentional semantic changes
## Current Project Direction
For this repository specifically, current review should assume:
- single-instance deployment is the primary target unless stated otherwise
- runtime semantics may intentionally differ from older Redis and asynq behavior when the new SQLite design is deliberate
- any such semantic difference must be explained in merge or review notes
+208
View File
@@ -0,0 +1,208 @@
# Review
## Scope
This review checks `dev-runtime` against `dev` with the following standards:
- Do not accidentally remove or override teammate payment capabilities already merged into `dev`.
- Do not submit local test artifacts or dirty files.
- Confirm Redis -> SQLite replacement is functionally complete for the intended single-instance target.
- Judge code quality, elegance, robustness, and maintainability.
- Check for real bugs, mojibake, and redundant code.
- Compare current quality with the previous `dev` implementation.
These standards are also the default development constraints for future work:
- no invalid tests
- no mirror tests
- prefer behavior-based tests over implementation-shaped tests
- explain intentional semantic changes in review notes
- verify dirty local artifacts are excluded before commit
- think about teammate-regression risk before coding, not only after coding
## Files That Must Not Be Committed
These local SQLite artifacts must never be committed:
- `src/epusdt.db`
- `src/epusdt.db-shm`
- `src/epusdt.db-wal`
They are now excluded by `.gitignore`.
## Teammate Capabilities Still Preserved
The following teammate payment capabilities are still present:
- multi-currency order model: `src/model/mdb/orders_mdb.go`
- wallet address model: `src/model/mdb/wallet_address_mdb.go`
- native `TRX` transfer detection: `src/model/service/task_service.go`
- `TRC20 USDT` detection: `src/model/service/task_service.go`
- dynamic exchange-rate path: `src/model/service/order_service.go`, `src/config/config.go`
- `TRON_GRID_API_KEY` support: `src/config/config.go`
Conclusion: teammate core payment capabilities were not accidentally removed.
## Intentional Architecture Changes
These are intentional refactors, not accidental overrides:
- Redis runtime removed: `src/model/dao/rdb.go`
- asynq handlers removed: `src/mq/handle/callback_queue.go`, `src/mq/handle/order_expiration_queue.go`
- SQLite runtime lock introduced: `src/model/mdb/transaction_lock_mdb.go`
- SQLite scheduler introduced: `src/mq/queue.go`, `src/mq/worker.go`
- reservation moved from Redis key to SQLite unique constraint: `src/model/data/order_data.go`
Conclusion: this branch keeps payment capability but changes the runtime implementation.
## Previously Regressed Parts That Are Now Restored
The following regressions have been restored in the current working tree:
- rich Telegram payment notification template: `src/model/service/task_service.go`
- key chain-scan observability logs: `src/model/service/task_service.go`
- Chinese checkout error message: `src/model/service/pay_service.go`
- more reliable Telegram add-wallet flow: `src/telegram/handle.go`
## Functional Completion
For the single-instance target, the Redis -> SQLite replacement is complete across:
- order creation
- amount reservation
- chain scan matching
- payment success processing
- order expiration
- callback scheduling
- Telegram notification
Conclusion: feature work is complete for the intended single-instance model.
## Mojibake Check
No real mojibake was found in the source files checked during review.
Files spot-checked:
- `src/telegram/handle.go`
- `src/model/service/task_service.go`
- `src/model/service/pay_service.go`
Note: some terminals may render UTF-8 Chinese poorly, but the source content itself is fine.
## Redundant Code Check
No obvious redundant business code was found.
Recent helper additions are justified:
- `PendingCallbackOrder`
- `expirableOrder`
- `configureSQLite`
Conclusion: no clear "fix by duplication" smell was found.
## Bug Assessment
No confirmed blocking business bug is currently visible in the reviewed code path.
Important risk that still needs attention:
- SQLite main DB can still hit `SQLITE_BUSY` under concurrent polling and writes.
This risk is now being mitigated by:
- `src/model/dao/sqlite_config.go`
- `src/model/dao/mdb_sqlite.go`
- `src/model/dao/runtime_sqlite.go`
- lightweight busy retry in `src/mq/worker.go`
Conclusion: no confirmed blocking functional bug, but SQLite concurrency remains a real operational boundary that should still be watched in runtime.
## Robustness Assessment
Strengths:
- conditional state transitions prevent reviving expired orders
- reservation is now a persistent unique constraint instead of a temporary Redis key
- callback state is durable across process restarts
- behavior tests exist for key paths:
- `src/model/service/order_service_test.go`
- `src/mq/worker_test.go`
- callback polling now has a small targeted retry for transient SQLite busy errors
Remaining limits:
- robust for single-instance deployment
- not a final design for multi-instance deployment
- SQLite contention should still be watched in real runtime
Conclusion: robust enough for single-instance use, not a direct multi-instance design.
## Elegance Assessment
Strengths:
- runtime responsibility is cleaner with SQLite reservation + scheduler
- logging responsibility is split more clearly:
- `log_level`
- `http_access_log`
- `sql_debug`
- Telegram input flow is more practical for real clients
- SQLite tuning is extracted into one helper
Tradeoffs:
- callback semantics are no longer the same as the old immediate-asynq enqueue path
- current branch still has local uncommitted fixes that must be organized into a clean commit set
Conclusion: elegant for the chosen single-instance direction, but must be explained clearly when merging.
## Quality Compared With Previous Code
Overall judgment: quality did not go down; it improved in the single-instance direction.
Improvements:
- stronger state-machine constraints
- better log control
- more durable runtime reservation
- better Telegram interaction resilience
- added behavior tests
- added restart-oriented callback recovery coverage
- reduced scheduler fragility under transient SQLite busy errors
Changed semantics rather than degradation:
- merchant callback is now polled by SQLite scheduler instead of immediate asynq enqueue
Conclusion: quality is not lower, but runtime semantics changed and must be stated explicitly.
## Self Score
- functional completeness: 9.1/10
- engineering implementation: 8.9/10
- robustness: 8.7/10
- elegance: 9.0/10
- commit readiness: 8.8/10
Overall self score: 8.9/10
Main deductions:
- current uncommitted fixes still need to be organized into a clean commit
- SQLite single-file concurrency boundary still exists
- callback semantics differ from `dev-payment` and require clear communication
## Final Conclusion
This code can be submitted, but only if all of the following are done:
1. Do not commit local SQLite artifacts.
2. Include the current uncommitted fixes in a clean, intentional commit set.
3. Explain clearly in merge notes:
- what teammate capabilities were preserved
- what runtime behavior was intentionally refactored
- what semantics changed and why
If those three conditions are met, this is not a hidden override of teammate code. It is a reviewed and documented refactor.
+3 -1
View File
@@ -1,6 +1,8 @@
app_name=epusdt
app_uri=https://dujiaoka.com
app_debug=false
log_level=info
http_access_log=false
sql_debug=false
http_listen=:8000
static_path=/static
+10 -15
View File
@@ -22,8 +22,8 @@ import (
var httpCmd = &cobra.Command{
Use: "http",
Short: "http服务",
Long: "http服务相关命令",
Short: "http service",
Long: "http service commands",
Run: func(cmd *cobra.Command, args []string) {
},
}
@@ -34,8 +34,8 @@ func init() {
var startCmd = &cobra.Command{
Use: "start",
Short: "启动",
Long: "启动http服务",
Short: "start",
Long: "start http service",
Run: func(cmd *cobra.Command, args []string) {
bootstrap.InitApp()
printBanner()
@@ -48,23 +48,22 @@ func HttpServerStart() {
e := echo.New()
e.HideBanner = true
e.HTTPErrorHandler = customHTTPErrorHandler
// 中间件注册
MiddlewareRegister(e)
// 路由注册
route.RegisterRoute(e)
// 静态目录注册
e.Static(config.StaticPath, "static")
e.Static(config.StaticPath, config.StaticFilePath)
httpListen := viper.GetString("http_listen")
go func() {
if err = e.Start(httpListen); err != http.ErrServerClosed {
log.Sugar.Error(err)
}
}()
// Wait for interrupt signal to gracefully shutdown the server with a timeout of 10 seconds.
// Use a buffered channel to avoid missing signals as recommended for signal.Notify
quit := make(chan os.Signal, 1)
signal.Notify(quit, os.Interrupt, os.Kill)
<-quit
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if err = e.Shutdown(ctx); err != nil {
@@ -72,16 +71,13 @@ func HttpServerStart() {
}
}
// MiddlewareRegister 中间件注册
func MiddlewareRegister(e *echo.Echo) {
if config.AppDebug {
e.Debug = true
if config.HTTPAccessLog {
e.Use(echoMiddleware.Logger())
}
e.Use(middleware.RequestUUID())
}
// customHTTPErrorHandler 默认消息提示
func customHTTPErrorHandler(err error, e echo.Context) {
code := http.StatusInternalServerError
msg := "server error"
@@ -99,5 +95,4 @@ func customHTTPErrorHandler(err error, e echo.Context) {
resp.Message = he.Msg
}
_ = e.JSON(http.StatusOK, resp)
return
}
+10 -1
View File
@@ -1,6 +1,11 @@
package command
import "github.com/spf13/cobra"
import (
"github.com/assimon/luuu/config"
"github.com/spf13/cobra"
)
var configPath string
var rootCmd = &cobra.Command{}
@@ -9,5 +14,9 @@ func Execute() error {
}
func init() {
rootCmd.PersistentFlags().StringVar(&configPath, "config", "", "path to .env or directory containing .env")
rootCmd.PersistentPreRun = func(cmd *cobra.Command, args []string) {
config.SetConfigPath(configPath)
}
rootCmd.AddCommand(httpCmd)
}
+110 -10
View File
@@ -1,6 +1,7 @@
package config
import (
"errors"
"fmt"
"log"
"net/url"
@@ -15,11 +16,14 @@ import (
)
var (
AppDebug bool
HTTPAccessLog bool
SQLDebug bool
LogLevel string
MysqlDns string
RuntimePath string
LogSavePath string
StaticPath string
StaticFilePath string
TgBotToken string
TgProxy string
TgManage int64
@@ -29,23 +33,33 @@ var (
BuildVersion = "0.0.0-dev"
BuildCommit = "none"
BuildDate = "unknown"
configRootPath string
explicitConfigPath string
)
func SetConfigPath(path string) {
explicitConfigPath = strings.TrimSpace(path)
}
func Init() {
viper.AddConfigPath("./")
viper.SetConfigFile(".env")
err := viper.ReadInConfig()
configPath, err := resolveConfigFilePath()
if err != nil {
panic(err)
}
gwd, err := os.Getwd()
configRootPath = filepath.Dir(configPath)
viper.SetConfigFile(configPath)
err = viper.ReadInConfig()
if err != nil {
panic(err)
}
AppDebug = viper.GetBool("app_debug")
StaticPath = viper.GetString("static_path")
RuntimePath = fmt.Sprintf("%s%s", gwd, viper.GetString("runtime_root_path"))
LogSavePath = fmt.Sprintf("%s%s", RuntimePath, viper.GetString("log_save_path"))
HTTPAccessLog = viper.GetBool("http_access_log")
SQLDebug = viper.GetBool("sql_debug")
LogLevel = normalizeLogLevel(viper.GetString("log_level"))
StaticPath = normalizeStaticURLPath(viper.GetString("static_path"))
StaticFilePath = filepath.Join(configRootPath, strings.TrimPrefix(StaticPath, "/"))
RuntimePath = resolvePathFromBase(configRootPath, viper.GetString("runtime_root_path"), filepath.Join(configRootPath, "runtime"))
LogSavePath = resolvePathFromBase(RuntimePath, viper.GetString("log_save_path"), filepath.Join(RuntimePath, "logs"))
mustMkdir(RuntimePath)
mustMkdir(LogSavePath)
MysqlDns = fmt.Sprintf("%s:%s@tcp(%s)/%s?charset=utf8mb4&parseTime=True&loc=Local",
@@ -67,6 +81,79 @@ func mustMkdir(path string) {
}
}
func normalizeLogLevel(level string) string {
switch strings.ToLower(strings.TrimSpace(level)) {
case "debug", "info", "warn", "error":
return strings.ToLower(strings.TrimSpace(level))
default:
return "info"
}
}
func normalizeStaticURLPath(path string) string {
path = strings.TrimSpace(path)
if path == "" || path == "/" {
return "/static"
}
if !strings.HasPrefix(path, "/") {
path = "/" + path
}
return path
}
func resolvePathFromBase(basePath string, path string, fallback string) string {
path = strings.TrimSpace(path)
if path == "" {
return fallback
}
if filepath.IsAbs(path) {
return path
}
path = strings.TrimPrefix(path, "/")
path = strings.TrimPrefix(path, "\\")
return filepath.Join(basePath, filepath.FromSlash(path))
}
func resolveConfigFilePath() (string, error) {
if explicitConfigPath != "" {
return normalizeConfiguredPath(explicitConfigPath)
}
if envPath := strings.TrimSpace(os.Getenv("EPUSDT_CONFIG")); envPath != "" {
return normalizeConfiguredPath(envPath)
}
return normalizeConfiguredPath(".env")
}
func normalizeConfiguredPath(input string) (string, error) {
path := strings.TrimSpace(input)
if path == "" {
path = ".env"
}
if !filepath.IsAbs(path) {
cwd, err := os.Getwd()
if err != nil {
return "", err
}
path = filepath.Join(cwd, path)
}
info, err := os.Stat(path)
if err == nil && info.IsDir() {
path = filepath.Join(path, ".env")
info, err = os.Stat(path)
}
if err != nil {
if errors.Is(err, os.ErrNotExist) {
return "", fmt.Errorf("config file not found: %s", path)
}
return "", err
}
if info.IsDir() {
return "", fmt.Errorf("config path must point to a file, got directory: %s", path)
}
return path, nil
}
func GetAppVersion() string {
return BuildVersion
}
@@ -194,7 +281,20 @@ func GetRuntimeSqlitePath() string {
if filepath.IsAbs(filename) {
return filename
}
return filepath.Join(RuntimePath, filename)
filename = strings.TrimPrefix(strings.TrimPrefix(filename, "/"), "\\")
return filepath.Join(RuntimePath, filepath.FromSlash(filename))
}
func GetPrimarySqlitePath() string {
filename := strings.TrimSpace(viper.GetString("sqlite_database_filename"))
if filename == "" {
return filepath.Join(configRootPath, "epusdt.db")
}
if filepath.IsAbs(filename) {
return filename
}
filename = strings.TrimPrefix(strings.TrimPrefix(filename, "/"), "\\")
return filepath.Join(configRootPath, filepath.FromSlash(filename))
}
func GetQueueConcurrency() int {
+118
View File
@@ -0,0 +1,118 @@
package config
import (
"os"
"path/filepath"
"testing"
)
func TestNormalizeConfiguredPathUsesExplicitFile(t *testing.T) {
t.Helper()
root := t.TempDir()
configPath := filepath.Join(root, "custom.env")
if err := os.WriteFile(configPath, []byte("app_name=test\n"), 0o644); err != nil {
t.Fatalf("write config: %v", err)
}
got, err := normalizeConfiguredPath(configPath)
if err != nil {
t.Fatalf("normalize explicit file: %v", err)
}
if got != configPath {
t.Fatalf("config path = %s, want %s", got, configPath)
}
}
func TestNormalizeConfiguredPathUsesExplicitDirectory(t *testing.T) {
t.Helper()
root := t.TempDir()
configPath := filepath.Join(root, ".env")
if err := os.WriteFile(configPath, []byte("app_name=test\n"), 0o644); err != nil {
t.Fatalf("write config: %v", err)
}
got, err := normalizeConfiguredPath(root)
if err != nil {
t.Fatalf("normalize explicit directory: %v", err)
}
if got != configPath {
t.Fatalf("config path = %s, want %s", got, configPath)
}
}
func TestResolveConfigFilePathUsesCurrentDirectoryByDefault(t *testing.T) {
t.Helper()
oldCwd, err := os.Getwd()
if err != nil {
t.Fatalf("get cwd: %v", err)
}
defer func() { _ = os.Chdir(oldCwd) }()
root := t.TempDir()
configPath := filepath.Join(root, ".env")
if err := os.WriteFile(configPath, []byte("app_name=test\n"), 0o644); err != nil {
t.Fatalf("write config: %v", err)
}
if err := os.Chdir(root); err != nil {
t.Fatalf("chdir: %v", err)
}
t.Setenv("EPUSDT_CONFIG", "")
SetConfigPath("")
got, err := resolveConfigFilePath()
if err != nil {
t.Fatalf("resolve config path: %v", err)
}
if got != configPath {
t.Fatalf("config path = %s, want %s", got, configPath)
}
}
func TestResolveConfigFilePathPrefersExplicitOverEnv(t *testing.T) {
t.Helper()
oldCwd, err := os.Getwd()
if err != nil {
t.Fatalf("get cwd: %v", err)
}
defer func() { _ = os.Chdir(oldCwd) }()
root := t.TempDir()
if err := os.Chdir(root); err != nil {
t.Fatalf("chdir: %v", err)
}
envDir := filepath.Join(root, "from-env")
if err := os.MkdirAll(envDir, 0o755); err != nil {
t.Fatalf("mkdir env dir: %v", err)
}
envPath := filepath.Join(envDir, ".env")
if err := os.WriteFile(envPath, []byte("app_name=env\n"), 0o644); err != nil {
t.Fatalf("write env config: %v", err)
}
flagDir := filepath.Join(root, "from-flag")
if err := os.MkdirAll(flagDir, 0o755); err != nil {
t.Fatalf("mkdir flag dir: %v", err)
}
flagPath := filepath.Join(flagDir, ".env")
if err := os.WriteFile(flagPath, []byte("app_name=flag\n"), 0o644); err != nil {
t.Fatalf("write flag config: %v", err)
}
t.Setenv("EPUSDT_CONFIG", envDir)
SetConfigPath(flagDir)
defer SetConfigPath("")
got, err := resolveConfigFilePath()
if err != nil {
t.Fatalf("resolve config path: %v", err)
}
if got != flagPath {
t.Fatalf("config path = %s, want %s", got, flagPath)
}
}
+2 -2
View File
@@ -1,13 +1,13 @@
package comm
import (
"fmt"
"github.com/assimon/luuu/config"
"github.com/assimon/luuu/model/response"
"github.com/assimon/luuu/model/service"
"github.com/labstack/echo/v4"
"html/template"
"net/http"
"path/filepath"
)
// CheckoutCounter 收银台
@@ -17,7 +17,7 @@ func (c *BaseCommController) CheckoutCounter(ctx echo.Context) (err error) {
if err != nil {
return ctx.String(http.StatusOK, err.Error())
}
tmpl, err := template.ParseFiles(fmt.Sprintf(".%s/%s", config.StaticPath, "index.html"))
tmpl, err := template.ParseFiles(filepath.Join(config.StaticFilePath, "index.html"))
if err != nil {
return ctx.String(http.StatusOK, err.Error())
}
+6 -1
View File
@@ -7,8 +7,10 @@ import (
"github.com/assimon/luuu/config"
"github.com/assimon/luuu/model/dao"
"github.com/assimon/luuu/model/mdb"
appLog "github.com/assimon/luuu/util/log"
"github.com/libtnb/sqlite"
"github.com/spf13/viper"
"go.uber.org/zap"
"gorm.io/gorm"
"gorm.io/gorm/logger"
"gorm.io/gorm/schema"
@@ -27,8 +29,11 @@ func SetupTestDatabases(t testing.TB) func() {
viper.Set("queue_poll_interval_ms", 50)
viper.Set("api_auth_token", "test-token")
config.AppDebug = false
config.HTTPAccessLog = false
config.SQLDebug = false
config.LogLevel = "error"
config.UsdtRate = 0
appLog.Sugar = zap.NewNop().Sugar()
mainDB := mustOpenSQLite(t, filepath.Join(t.TempDir(), "main.db"))
runtimeDB := mustOpenSQLite(t, filepath.Join(t.TempDir(), "runtime.db"))
+1 -1
View File
@@ -29,7 +29,7 @@ func MysqlInit() error {
// panic(err)
return err
}
if config.AppDebug {
if config.SQLDebug {
Mdb = Mdb.Debug()
}
sqlDB, err := Mdb.DB()
+1 -1
View File
@@ -40,7 +40,7 @@ func PostgreSQLInit() error {
return err
}
if config.AppDebug {
if config.SQLDebug {
Mdb = Mdb.Debug()
}
sqlDB, err := Mdb.DB()
+7 -16
View File
@@ -1,6 +1,7 @@
package dao
import (
"os"
"path/filepath"
"github.com/assimon/luuu/config"
@@ -16,9 +17,10 @@ import (
// SqliteInit 数据库初始化
func SqliteInit() error {
var err error
dbFilename := "./conf/.db"
if dbfile := viper.GetString("sqlite_database_filename"); len(dbfile) > 0 {
dbFilename = filepath.Base(dbfile)
dbFilename := config.GetPrimarySqlitePath()
if err = os.MkdirAll(filepath.Dir(dbFilename), 0o755); err != nil {
color.Red.Printf("[store_db] sqlite mkdir err=%s\n", err)
return err
}
color.Green.Printf("[store_db] sqlite filename: %s\n", dbFilename)
Mdb, err = openDB(dbFilename, &gorm.Config{
@@ -33,22 +35,11 @@ func SqliteInit() error {
// panic(err)
return err
}
if config.AppDebug {
if config.SQLDebug {
Mdb = Mdb.Debug()
}
sqlDB, err := Mdb.DB()
if err != nil {
color.Red.Printf("[store_db] sqlite get DB,err=%s\n", err)
// panic(err)
return err
}
// sqlDB.SetMaxIdleConns(viper.GetInt("sqlite_max_idle_conns"))
// sqlDB.SetMaxOpenConns(viper.GetInt("sqlite_max_open_conns"))
// sqlDB.SetConnMaxLifetime(time.Hour * time.Duration(viper.GetInt("sqlite_max_life_time")))
err = sqlDB.Ping()
if err != nil {
if _, err = configureSQLite(Mdb, 1); err != nil {
color.Red.Printf("[store_db] sqlite connDB err:%s", err.Error())
// panic(err)
return err
}
log.Sugar.Debug("[store_db] sqlite connDB success")
+1 -19
View File
@@ -27,12 +27,6 @@ func RuntimeInit() error {
return err
}
sqlDB, err := RuntimeDB.DB()
if err != nil {
color.Red.Printf("[runtime_db] sqlite get DB,err=%s\n", err)
return err
}
concurrency := config.GetQueueConcurrency()
if concurrency < 2 {
concurrency = 2
@@ -40,19 +34,7 @@ func RuntimeInit() error {
if concurrency > 16 {
concurrency = 16
}
sqlDB.SetMaxOpenConns(concurrency)
sqlDB.SetMaxIdleConns(1)
if err = RuntimeDB.Exec("PRAGMA journal_mode=WAL;").Error; err != nil {
return err
}
if err = RuntimeDB.Exec("PRAGMA synchronous=NORMAL;").Error; err != nil {
return err
}
if err = RuntimeDB.Exec("PRAGMA busy_timeout=5000;").Error; err != nil {
return err
}
if err = sqlDB.Ping(); err != nil {
if _, err = configureSQLite(RuntimeDB, concurrency); err != nil {
color.Red.Printf("[runtime_db] sqlite connDB err:%s", err.Error())
return err
}
+35
View File
@@ -0,0 +1,35 @@
package dao
import (
"database/sql"
"gorm.io/gorm"
)
func configureSQLite(db *gorm.DB, maxOpenConns int) (*sql.DB, error) {
sqlDB, err := db.DB()
if err != nil {
return nil, err
}
if maxOpenConns <= 0 {
maxOpenConns = 1
}
sqlDB.SetMaxOpenConns(maxOpenConns)
sqlDB.SetMaxIdleConns(1)
if err := db.Exec("PRAGMA journal_mode=WAL;").Error; err != nil {
return nil, err
}
if err := db.Exec("PRAGMA synchronous=NORMAL;").Error; err != nil {
return nil, err
}
if err := db.Exec("PRAGMA busy_timeout=5000;").Error; err != nil {
return nil, err
}
if err := sqlDB.Ping(); err != nil {
return nil, err
}
return sqlDB, nil
}
+12 -3
View File
@@ -8,6 +8,7 @@ import (
"github.com/assimon/luuu/model/dao"
"github.com/assimon/luuu/model/mdb"
"github.com/assimon/luuu/model/request"
"github.com/dromara/carbon/v2"
"github.com/shopspring/decimal"
"gorm.io/gorm"
"gorm.io/gorm/clause"
@@ -15,6 +16,13 @@ import (
var ErrTransactionLocked = errors.New("transaction amount is already locked")
type PendingCallbackOrder struct {
TradeId string `gorm:"column:trade_id"`
CallbackNum int `gorm:"column:callback_num"`
CallBackConfirm int `gorm:"column:callback_confirm"`
UpdatedAt carbon.Time `gorm:"column:updated_at"`
}
func normalizeLockAmount(amount float64) (int64, string) {
value := decimal.NewFromFloat(amount).Round(2)
return value.Shift(2).IntPart(), value.StringFixed(2)
@@ -63,10 +71,11 @@ func OrderSuccessWithTransaction(tx *gorm.DB, req *request.OrderProcessingReques
return result.RowsAffected > 0, result.Error
}
// GetPendingCallbackOrders returns orders that still need callback delivery.
func GetPendingCallbackOrders(maxRetry int, limit int) ([]mdb.Orders, error) {
var orders []mdb.Orders
// GetPendingCallbackOrders returns the minimal callback scheduling state.
func GetPendingCallbackOrders(maxRetry int, limit int) ([]PendingCallbackOrder, error) {
var orders []PendingCallbackOrder
query := dao.Mdb.Model(&mdb.Orders{}).
Select("trade_id", "callback_num", "callback_confirm", "updated_at").
Where("callback_num <= ?", maxRetry).
Where("callback_confirm = ?", mdb.CallBackConfirmNo).
Where("status = ?", mdb.StatusPaySuccess).
+1 -1
View File
@@ -16,7 +16,7 @@ func GetCheckoutCounterByTradeId(tradeId string) (*response.CheckoutCounterRespo
return nil, err
}
if orderInfo.ID <= 0 || orderInfo.Status != mdb.StatusWaitPay {
return nil, errors.New("pending order does not exist or has expired")
return nil, errors.New("不存在待支付订单或已过期")
}
resp := &response.CheckoutCounterResponse{
+36 -6
View File
@@ -8,8 +8,8 @@ import (
"strings"
"sync"
tron "github.com/assimon/luuu/crypto"
"github.com/assimon/luuu/config"
tron "github.com/assimon/luuu/crypto"
"github.com/assimon/luuu/model/data"
"github.com/assimon/luuu/model/mdb"
"github.com/assimon/luuu/model/request"
@@ -73,7 +73,14 @@ func checkTrxTransfers(address string, wg *sync.WaitGroup) {
panic("TRX API response indicates failure")
}
for i, transfer := range gjson.GetBytes(resp.Body(), "data").Array() {
transfers := gjson.GetBytes(resp.Body(), "data").Array()
if len(transfers) == 0 {
log.Sugar.Debugf("[TRX][%s] no transfer records found", address)
return
}
log.Sugar.Debugf("[TRX][%s] fetched %d transfer records", address, len(transfers))
for i, transfer := range transfers {
if transfer.Get("raw_data.contract.0.type").String() != "TransferContract" {
continue
}
@@ -108,8 +115,10 @@ func checkTrxTransfers(address string, wg *sync.WaitGroup) {
panic(err)
}
if tradeID == "" {
log.Sugar.Debugf("[TRX][%s] skip unmatched tx hash=%s amount=%.2f", address, txID, amount)
continue
}
log.Sugar.Infof("[TRX][%s] matched trade_id=%s hash=%s amount=%.2f", address, tradeID, txID, amount)
order, err := data.GetOrderInfoByTradeId(tradeID)
if err != nil {
@@ -139,6 +148,7 @@ func checkTrxTransfers(address string, wg *sync.WaitGroup) {
}
sendPaymentNotification(order)
log.Sugar.Infof("[TRX][%s] payment processed trade_id=%s hash=%s", address, tradeID, txID)
}
}
@@ -174,7 +184,14 @@ func checkTrc20Transfers(address string, wg *sync.WaitGroup) {
panic("TRC20 API response indicates failure")
}
for i, transfer := range gjson.GetBytes(resp.Body(), "data").Array() {
transfers := gjson.GetBytes(resp.Body(), "data").Array()
if len(transfers) == 0 {
log.Sugar.Debugf("[TRC20][%s] no transfer records found", address)
return
}
log.Sugar.Debugf("[TRC20][%s] fetched %d transfer records", address, len(transfers))
for i, transfer := range transfers {
if transfer.Get("token_info.address").String() != TRC20_USDT_ID {
continue
}
@@ -200,8 +217,10 @@ func checkTrc20Transfers(address string, wg *sync.WaitGroup) {
panic(err)
}
if tradeID == "" {
log.Sugar.Debugf("[TRC20][%s] skip unmatched tx hash=%s amount=%.2f", address, txID, amount)
continue
}
log.Sugar.Infof("[TRC20][%s] matched trade_id=%s hash=%s amount=%.2f", address, tradeID, txID, amount)
order, err := data.GetOrderInfoByTradeId(tradeID)
if err != nil {
@@ -231,18 +250,29 @@ func checkTrc20Transfers(address string, wg *sync.WaitGroup) {
}
sendPaymentNotification(order)
log.Sugar.Infof("[TRC20][%s] payment processed trade_id=%s hash=%s", address, tradeID, txID)
}
}
func sendPaymentNotification(order *mdb.Orders) {
msg := fmt.Sprintf(
"Payment received\nTrade ID: %s\nOrder ID: %s\nOrder Amount: %.2f %s\nReceived Amount: %.2f %s\nAddress: %s\nCreated At: %s\nPaid At: %s",
order.TradeId,
order.OrderId,
"🎉 <b>收款成功通知</b>\n\n"+
"💰 <b>金额信息</b>\n"+
"├ 订单金额:<code>%.2f %s</code>\n"+
"└ 实际到账:<code>%.2f %s</code>\n\n"+
"📋 <b>订单信息</b>\n"+
"├ 交易号:<code>%s</code>\n"+
"├ 订单号:<code>%s</code>\n"+
"└ 钱包地址:<code>%s</code>\n\n"+
"⏰ <b>时间信息</b>\n"+
"├ 创建时间:%s\n"+
"└ 支付时间:%s",
order.Amount,
strings.ToUpper(order.Currency),
order.ActualAmount,
strings.ToUpper(order.Token),
order.TradeId,
order.OrderId,
order.ReceiveAddress,
order.CreatedAt.ToDateTimeString(),
carbon.Now().ToDateTimeString(),
+54 -11
View File
@@ -3,6 +3,7 @@ package mq
import (
"errors"
"net/http"
"strings"
"time"
"github.com/assimon/luuu/config"
@@ -17,6 +18,16 @@ import (
const batchSize = 100
const sqliteBusyRetryAttempts = 3
type expirableOrder struct {
ID uint64 `gorm:"column:id"`
TradeId string `gorm:"column:trade_id"`
ReceiveAddress string `gorm:"column:receive_address"`
Token string `gorm:"column:token"`
ActualAmount float64 `gorm:"column:actual_amount"`
}
func runOrderExpirationLoop() {
runLoop("order_expiration", processExpiredOrders)
}
@@ -51,13 +62,16 @@ func safeRun(name string, fn func()) {
func processExpiredOrders() {
expirationCutoff := time.Now().Add(-config.GetOrderExpirationTimeDuration())
for {
var orders []mdb.Orders
err := dao.Mdb.Model(&mdb.Orders{}).
var orders []expirableOrder
err := withSQLiteBusyRetry(func() error {
return dao.Mdb.Model(&mdb.Orders{}).
Select("id", "trade_id", "receive_address", "token", "actual_amount").
Where("status = ?", mdb.StatusWaitPay).
Where("created_at <= ?", expirationCutoff).
Order("id asc").
Limit(batchSize).
Find(&orders).Error
})
if err != nil {
log.Sugar.Errorf("[mq] query expired orders failed: %v", err)
return
@@ -88,7 +102,12 @@ func processExpiredOrders() {
func dispatchPendingCallbacks() {
maxRetry := config.GetOrderNoticeMaxRetry()
orders, err := data.GetPendingCallbackOrders(maxRetry, batchSize)
var orders []data.PendingCallbackOrder
err := withSQLiteBusyRetry(func() error {
var innerErr error
orders, innerErr = data.GetPendingCallbackOrders(maxRetry, batchSize)
return innerErr
})
if err != nil {
log.Sugar.Errorf("[mq] query callback orders failed: %v", err)
return
@@ -99,29 +118,30 @@ func dispatchPendingCallbacks() {
if !isCallbackDue(&order, now, maxRetry) {
continue
}
if _, loaded := callbackInflight.LoadOrStore(order.TradeId, struct{}{}); loaded {
tradeID := order.TradeId
if _, loaded := callbackInflight.LoadOrStore(tradeID, struct{}{}); loaded {
continue
}
select {
case callbackLimiter <- struct{}{}:
go processCallback(order)
go processCallback(tradeID)
default:
callbackInflight.Delete(order.TradeId)
callbackInflight.Delete(tradeID)
return
}
}
}
func processCallback(order mdb.Orders) {
func processCallback(tradeID string) {
defer func() {
<-callbackLimiter
callbackInflight.Delete(order.TradeId)
callbackInflight.Delete(tradeID)
}()
freshOrder, err := data.GetOrderInfoByTradeId(order.TradeId)
freshOrder, err := data.GetOrderInfoByTradeId(tradeID)
if err != nil {
log.Sugar.Errorf("[mq] reload callback order failed, trade_id=%s, err=%v", order.TradeId, err)
log.Sugar.Errorf("[mq] reload callback order failed, trade_id=%s, err=%v", tradeID, err)
return
}
if freshOrder.ID <= 0 || freshOrder.Status != mdb.StatusPaySuccess || freshOrder.CallBackConfirm != mdb.CallBackConfirmNo {
@@ -180,7 +200,30 @@ func cleanupExpiredTransactionLocks() {
}
}
func isCallbackDue(order *mdb.Orders, now time.Time, maxRetry int) bool {
func withSQLiteBusyRetry(fn func() error) error {
var err error
for attempt := 1; attempt <= sqliteBusyRetryAttempts; attempt++ {
err = fn()
if err == nil {
return nil
}
if !isSQLiteBusyError(err) || attempt == sqliteBusyRetryAttempts {
return err
}
time.Sleep(time.Duration(attempt*25) * time.Millisecond)
}
return err
}
func isSQLiteBusyError(err error) bool {
if err == nil {
return false
}
msg := strings.ToLower(err.Error())
return strings.Contains(msg, "database is locked") || strings.Contains(msg, "sqlite_busy")
}
func isCallbackDue(order *data.PendingCallbackOrder, now time.Time, maxRetry int) bool {
if order.CallBackConfirm != mdb.CallBackConfirmNo {
return false
}
+72
View File
@@ -185,6 +185,78 @@ func TestDispatchPendingCallbacksHonorsBackoffAndPersistsSuccess(t *testing.T) {
}
}
func TestDispatchPendingCallbacksResumesRetryAfterRestart(t *testing.T) {
cleanup := testutil.SetupTestDatabases(t)
defer cleanup()
callbackLimiter = make(chan struct{}, 1)
callbackInflight = sync.Map{}
var requestCount int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
attempt := atomic.AddInt32(&requestCount, 1)
if attempt == 1 {
http.Error(w, "retry later", http.StatusInternalServerError)
return
}
_, _ = io.WriteString(w, "ok")
}))
defer server.Close()
order := &mdb.Orders{
TradeId: "trade_callback_restart",
OrderId: "order_callback_restart",
Amount: 1,
Currency: "CNY",
ActualAmount: 1,
ReceiveAddress: "wallet_restart",
Token: "USDT",
Status: mdb.StatusPaySuccess,
NotifyUrl: server.URL,
BlockTransactionId: "block_callback_restart",
CallbackNum: 0,
CallBackConfirm: mdb.CallBackConfirmNo,
}
if err := dao.Mdb.Create(order).Error; err != nil {
t.Fatalf("create callback order: %v", err)
}
dispatchPendingCallbacks()
waitFor(t, 3*time.Second, func() bool {
current, err := data.GetOrderInfoByTradeId(order.TradeId)
if err != nil || current.ID <= 0 {
return false
}
return current.CallBackConfirm == mdb.CallBackConfirmNo && current.CallbackNum == 1
})
if got := atomic.LoadInt32(&requestCount); got != 1 {
t.Fatalf("first callback request count = %d, want 1", got)
}
callbackLimiter = make(chan struct{}, 1)
callbackInflight = sync.Map{}
if err := dao.Mdb.Model(order).UpdateColumn("updated_at", time.Now().Add(-2*time.Second)).Error; err != nil {
t.Fatalf("age callback order for retry: %v", err)
}
dispatchPendingCallbacks()
waitFor(t, 3*time.Second, func() bool {
current, err := data.GetOrderInfoByTradeId(order.TradeId)
if err != nil || current.ID <= 0 {
return false
}
return current.CallBackConfirm == mdb.CallBackConfirmOk && current.CallbackNum == 2
})
if got := atomic.LoadInt32(&requestCount); got != 2 {
t.Fatalf("total callback request count = %d, want 2", got)
}
}
func waitFor(t *testing.T, timeout time.Duration, fn func() bool) {
t.Helper()
+66 -11
View File
@@ -2,6 +2,8 @@ package telegram
import (
"fmt"
"sync"
"time"
"github.com/assimon/luuu/model/data"
"github.com/assimon/luuu/model/mdb"
@@ -12,54 +14,87 @@ import (
const (
ReplayAddWallet = "请发给我一个合法的钱包地址"
pendingWalletAddressTTL = 5 * time.Minute
)
type pendingWalletAddressState struct {
RequestedAt time.Time
}
var pendingWalletAddressUsers sync.Map
func OnTextMessageHandle(c tb.Context) error {
if c.Message().ReplyTo.Text == ReplayAddWallet {
defer bots.Delete(c.Message().ReplyTo)
msgText := c.Message().Text
if !isValidTronAddress(msgText) {
_ = c.Send(fmt.Sprintf("钱包[%s]添加失败: 非Tron地址!", msgText))
msg := c.Message()
if msg == nil {
return nil
}
sender := c.Sender()
senderID := int64(0)
if sender != nil {
senderID = sender.ID
}
isReplyFlow := msg.ReplyTo != nil && msg.ReplyTo.Text == ReplayAddWallet
isPendingFlow := isWalletAddressPending(senderID)
if !isReplyFlow && !isPendingFlow {
return nil
}
if isReplyFlow {
defer bots.Delete(msg.ReplyTo)
}
msgText := msg.Text
if !isValidTronAddress(msgText) {
_ = c.Send(fmt.Sprintf("钱包 [%s] 添加失败:不是合法的 TRON 地址", msgText))
return nil
}
_, err := data.AddWalletAddress(msgText)
if err != nil {
return c.Send(err.Error())
}
_ = c.Send(fmt.Sprintf("钱包[%s]添加成功!", msgText))
pendingWalletAddressUsers.Delete(senderID)
_ = c.Send(fmt.Sprintf("钱包 [%s] 添加成功", msgText))
return WalletList(c)
}
return nil
}
func WalletList(c tb.Context) error {
wallets, err := data.GetAllWalletAddress()
if err != nil {
return err
}
var btnList [][]tb.InlineButton
for _, wallet := range wallets {
status := "已启用✅"
if wallet.Status == mdb.TokenStatusDisable {
status = "已禁用🚫"
}
var temp []tb.InlineButton
btnInfo := tb.InlineButton{
Unique: wallet.Address,
Text: fmt.Sprintf("%s [%s]", wallet.Address, status),
Data: strutil.MustString(wallet.ID),
}
bots.Handle(&btnInfo, WalletInfo)
btnList = append(btnList, append(temp, btnInfo))
btnList = append(btnList, []tb.InlineButton{btnInfo})
}
addBtn := tb.InlineButton{Text: "添加钱包地址", Unique: "AddWallet"}
bots.Handle(&addBtn, func(c tb.Context) error {
if sender := c.Sender(); sender != nil {
pendingWalletAddressUsers.Store(sender.ID, pendingWalletAddressState{RequestedAt: time.Now()})
}
return c.Send(ReplayAddWallet, &tb.ReplyMarkup{
ForceReply: true,
})
})
btnList = append(btnList, []tb.InlineButton{addBtn})
return c.EditOrSend("请点击钱包继续操作", &tb.ReplyMarkup{
return c.EditOrSend("请选择钱包继续操作", &tb.ReplyMarkup{
InlineKeyboard: btnList,
})
}
@@ -70,6 +105,7 @@ func WalletInfo(c tb.Context) error {
if err != nil {
return c.Send(err.Error())
}
enableBtn := tb.InlineButton{
Text: "启用",
Unique: "enableBtn",
@@ -89,10 +125,12 @@ func WalletInfo(c tb.Context) error {
Text: "返回",
Unique: "WalletList",
}
bots.Handle(&enableBtn, EnableWallet)
bots.Handle(&disableBtn, DisableWallet)
bots.Handle(&delBtn, DelWallet)
bots.Handle(&backBtn, WalletList)
return c.EditOrReply(tokenInfo.Address, &tb.ReplyMarkup{InlineKeyboard: [][]tb.InlineButton{
{
enableBtn,
@@ -140,3 +178,20 @@ func DelWallet(c tb.Context) error {
}
return WalletList(c)
}
func isWalletAddressPending(userID int64) bool {
if userID <= 0 {
return false
}
value, ok := pendingWalletAddressUsers.Load(userID)
if !ok {
return false
}
state, ok := value.(pendingWalletAddressState)
if !ok || time.Since(state.RequestedAt) > pendingWalletAddressTTL {
pendingWalletAddressUsers.Delete(userID)
return false
}
return true
}
+3 -2
View File
@@ -1,8 +1,9 @@
package http_client
import (
"github.com/go-resty/resty/v2"
"time"
"github.com/go-resty/resty/v2"
)
// GetHttpClient 获取请求客户端
@@ -13,6 +14,6 @@ func GetHttpClient(proxys ...string) *resty.Client {
proxy := proxys[0]
client.SetProxy(proxy)
}
client.SetTimeout(time.Second * 5)
client.SetTimeout(time.Second * 10)
return client
}
+32 -5
View File
@@ -7,27 +7,37 @@ import (
"github.com/spf13/viper"
"go.uber.org/zap"
"go.uber.org/zap/zapcore"
"os"
"time"
)
var Sugar *zap.SugaredLogger
func Init() {
writeSyncer := getLogWriter()
encoder := getEncoder()
core := zapcore.NewCore(encoder, writeSyncer, zapcore.DebugLevel)
level := getLogLevel()
core := zapcore.NewTee(
zapcore.NewCore(getConsoleEncoder(), getConsoleWriter(), level),
zapcore.NewCore(getFileEncoder(), getFileWriter(), level),
)
logger := zap.New(core, zap.AddCaller())
Sugar = logger.Sugar()
}
func getEncoder() zapcore.Encoder {
func getFileEncoder() zapcore.Encoder {
encoderConfig := zap.NewProductionEncoderConfig()
encoderConfig.EncodeTime = zapcore.ISO8601TimeEncoder
encoderConfig.EncodeLevel = zapcore.CapitalLevelEncoder
return zapcore.NewJSONEncoder(encoderConfig)
}
func getLogWriter() zapcore.WriteSyncer {
func getConsoleEncoder() zapcore.Encoder {
encoderConfig := zap.NewDevelopmentEncoderConfig()
encoderConfig.EncodeTime = zapcore.ISO8601TimeEncoder
encoderConfig.EncodeLevel = zapcore.CapitalColorLevelEncoder
return zapcore.NewConsoleEncoder(encoderConfig)
}
func getFileWriter() zapcore.WriteSyncer {
file := fmt.Sprintf("%s/log_%s.log",
config.LogSavePath,
time.Now().Format("20060102"))
@@ -40,3 +50,20 @@ func getLogWriter() zapcore.WriteSyncer {
}
return zapcore.AddSync(lumberJackLogger)
}
func getConsoleWriter() zapcore.WriteSyncer {
return zapcore.Lock(os.Stdout)
}
func getLogLevel() zapcore.Level {
switch config.LogLevel {
case "debug":
return zapcore.DebugLevel
case "warn":
return zapcore.WarnLevel
case "error":
return zapcore.ErrorLevel
default:
return zapcore.InfoLevel
}
}