Lua Archives - 第2頁共2頁 - 論野生技術&二次元

cidr.me地理位置查詢

on 2016 年 7 月 26 日

2 16318 轉為簡體

基於OpenResty ，MaxMind GeoIP數據庫和從bgp.he.net生成的ASN數據庫，因為沒有經緯度的需求所以沒有顯示。下文有源代碼的鏈接，如果需要可以自行修改加上經緯度或者將輸出變為JSON等。

這裡有chrome插件

使用方法

查詢當前IPv4地址

$ curl http://cidr.me/ip
x.x.x.x

查詢當前IPv6地址

$ curl http://ipv6.cidr.me/ip
x.x.x.x

查詢當前IP的PTR記錄（反向DNS），地理位置和ASN

$ curl http://cidr.me/
x.x.x.x
Country, City
ASN number

查詢當前IP的PTR記錄

$ curl http://cidr.me/rdns
x-x-x-x.com

查詢指定IP的地理位置

$ curl http://cidr.me/74.125.203.199
74.125.203.199 th-in-f199.1e100.net
United States, California, Mountain View
AS15169 Google Inc.

查詢域名的地理位置

$ curl http://cidr.me/www.google.com.hk
172.217.3.195 sea15s12-in-f3.1e100.net
United States, California, Mountain View
AS15169 Google LLC

2607:f8b0:400a:809::2003 sea15s12-in-x03.1e100.net
United States
AS15169 Google LLC

查詢域名的IP

$ curl http://cidr.me/www.google.com.hk/ip
74.125.203.199
2607:f8b0:400a:809::2003

查詢域名的IP和CNAME（如果存在）

$ curl http://cidr.me/www.youtube.com/dns
youtube-ui.l.google.com 172.217.3.174
172.217.3.206
216.58.193.78
216.58.217.46
2607:f8b0:400a:808::200e

源代碼

在這裡https://gist.github.com/fffonion/44e5fb59e2a8f0efba5c1965c6043584

需要ngx_http_geoip_module, echo-nginx-module, lua-nginx-module，安裝libgeoip，並將maxmind的舊版geoip數據庫放在/usr/share/GeoIP

通過bgp.he.net生成ASN數據庫

用 OpenResty 寫了一個 SNI 代理

on 2016 年 6 月 6 日

0 14012 轉為簡體

功能類似於dlundquist/sniproxy

推薦 OpenResty 加上 stream 模塊和 ngx_stream_lua_module 模塊。在 1.9.15.1 上測試通過。

SNI Proxy based on stream-lua-nginx-module
https://github.com/fffonion/lua-resty-sniproxy
18 forks.
83 stars.
2 open issues.

Recent commits:

doc(readme) fix example nginx configuration, Wangchong Zhou
release: 0.22, Wangchong Zhou
fix: proxy protocol on openresty 1.17.8, Wangchong Zhou
release: 0.21, Wangchong Zhou
proxy protocol support, Wangchong Zhou

示例配置：

stream {
    lua_resolver 8.8.8.8;
    init_worker_by_lua_block {
        sni_rules = { 
            ["www.google.com"] = {"www.google.com", 443},
            ["www.facebook.com"] = {"9.8.7.6", 443},
            ["twitter.com"] = {"1.2.3.4"},
            [".+.twitter.com"] = {nil, 443}
        }   
    }

    server {
            error_log /var/log/nginx/sniproxy-error.log error;
            listen 443;
            content_by_lua_block {
                    local sni = require("resty.sniproxy")
                    local sp = sni:new()
                    sp:run()
            }   
    }
}

stream {

lua_resolver 8.8.8.8;

init_worker_by_lua_block {

sni_rules = {

["www.google.com"] = {"www.google.com", 443},

["www.facebook.com"] = {"9.8.7.6", 443},

["twitter.com"] = {"1.2.3.4"},

[".+.twitter.com"] = {nil, 443}

}

server {

error_log /var/log/nginx/sniproxy-error.log error;

listen 443;

content_by_lua_block {

local sni = require("resty.sniproxy")

local sp = sni:new()

sp:run()

}

A Lua table sni_rules should be defined in the init_worker_by_lua_block directive.

The key can be either whole host name or regular expression. Use . for a default host name. If no entry is matched, connection will be closed.

The value is a table containing host name and port. If host is set to nil, the server_name in SNI will be used. If the port is not defined or set to nil, 443 will be used.

Rules are applied with the priority as its occurrence sequence in the table. In the example above, twitter.com will match the third rule rather than the fourth.

If the protocol version is less than TLSv1 (eg. SSLv3, SSLv2), connection will be closed, since SNI extension is not supported in these versions.

nginx 批量配置同步

on 2015 年 11 月 5 日

0 71473 轉為簡體

在編譯了lua-nginx-module的nginx上，可以方便地使用shared dict特性，在不reload配置文件的情況下實現配置同步。

由於shared dict使用一塊共享內存，因此所有worker均可讀寫，也就不存在一致性的問題。

使用shared dict

nginx/openresty的一些記錄

on 2015 年 10 月 16 日

28 41228 轉為簡體

日誌

屏蔽user-agent並屏蔽日誌

if ($http_user_agent ~* (^badbot/useragent$)) {
    rewrite (.*) /badbot break;
}

location = /badbot {
    access_log off;
    return 403;
}

if ($http_user_agent ~* (^badbot/useragent$)) {

rewrite (.*) /badbot break;

}

location = /badbot {

access_log off;

return 403;

}

不屏蔽user-agent（允許其訪問），但屏蔽日誌

server {
    set $is_spider 1;
    if ($http_user_agent ~* "bot|spider|Bot|Disqus|WebIndex|YunGuanCe") {
        set $is_spider 0;
    }
    access_log /var/log/nginx/access.log combined if=$is_spider;
}

server {

set $is_spider 1;

if ($http_user_agent ~* "bot|spider|Bot|Disqus|WebIndex|YunGuanCe") {

set $is_spider 0;

}

access_log /var/log/nginx/access.log combined if=$is_spider;

}

按uri屏蔽日誌（可以和上面的按user-agent用同一個變量來同時過濾uri和user-agent)

map $request $loggable {
    ~/favicon.ico 0;
    ~/images/* 0;
    ~/js/* 0;
    ~/css/* 0;
    default 1;
}

server {
    access_log /var/log/nginx/access.log combined if=$loggable;
}

map $request $loggable {

~/favicon.ico 0;

~/images/* 0;

~/js/* 0;

~/css/* 0;

default 1;

}

server {

access_log /var/log/nginx/access.log combined if=$loggable;

}

Header

按mime type設置緩存時間

map $sent_http_content_type $cacheable_types {
    "text/css"    "max-age=14400";
    "text/plain"  "max-age=86400";
    "image/jpeg"  "max-age=86400";
    "image/png"   "max-age=86400";
    "image/gif"   "max-age=86400";
    "image/x-icon" "max-age=86400";
    "application/x-7z-compressed" "max-age=864000";
    "application/x-javascript" "max-age=86400";
    "application/json" "max-age=86400";
    "application/x-bittorrent" "max-age=864000";
    default       "";
}

server {
    location / {
        root   "/var/www/html/";           
        add_header "Cache-Control" $cacheable_types;
    }
}

map $sent_http_content_type $cacheable_types {

"text/css" "max-age=14400";

"text/plain" "max-age=86400";

"image/jpeg" "max-age=86400";

"image/png" "max-age=86400";

"image/gif" "max-age=86400";

"image/x-icon" "max-age=86400";

"application/x-7z-compressed" "max-age=864000";

"application/x-javascript" "max-age=86400";

"application/json" "max-age=86400";

"application/x-bittorrent" "max-age=864000";

default "";

}

server {

location / {

root "/var/www/html/";

add_header "Cache-Control" $cacheable_types;

}

防攻擊

簡單的無狀態cookie challenge（需要lua-nginx-module)

crawlers塊中可以手動填寫要屏蔽的IP

將其中的s改成隨機字符串＋時間戳可以變成有狀態版本（需使用redis/memcached/shared memory存儲生成的隨機字符串）

將set-cookie改成通過js生成cookie可以變成javascript challenge，注意要在js里加上瀏覽器上下文判斷，如var cookie=location.protocol?cookie:””; 或者DOM操作

這裡有個更高級的輸驗證碼的示例

geo $crawlers{
    1.2.3.0/24 1;
    4.5.6.7 1;
    default 0;
}
server {
    location ~ \.php {
        set $_c 0;
        if ($http_user_agent ~* "bad.guy/123") {
            set $_c 1;
        }            
        if ($crawlers){
            set $_c 1;
        }
        if ($_c){
            access_log     off;
            access_by_lua_block {
                local expires = 90
                local s = ngx.time()
                local cob = tonumber(ngx.var.cookie_cob)
                if cob == nil then cob=0 end
                local coa = ngx.md5("saltsalt" .. ngx.var.remote_addr .. "saltsalt" .. cob)
                if s - cob > expires or ngx.var.cookie_coa ~= coa then
                    coa = ngx.md5("saltsalt" .. ngx.var.remote_addr .. "saltsalt" .. s)
                    ngx.header["Set-Cookie"] = {"coa=" .. coa .."; path=/; domain=.example.com; HttpOnly",
                        "cob=" .. s.."; path=/; domain=.example.com; HttpOnly"}
                    ngx.header["Content-Type"] = "text/html"
                    ngx.say("<meta http-equiv ='refresh' content='0'>");
                    ngx.exit(200)
                end
            }
        }
        # fastcgi_pass
    }
}

geo $crawlers{

1.2.3.0/24 1;

4.5.6.7 1;

default 0;

}

server {

location ~ \.php {

set $_c 0;

if ($http_user_agent ~* "bad.guy/123") {

set $_c 1;

}

if ($crawlers){

set $_c 1;

}

if ($_c){

access_log off;

access_by_lua_block {

local expires = 90

local s = ngx.time()

local cob = tonumber(ngx.var.cookie_cob)

if cob == nil then cob=0 end

local coa = ngx.md5("saltsalt" .. ngx.var.remote_addr .. "saltsalt" .. cob)

if s - cob > expires or ngx.var.cookie_coa ~= coa then

coa = ngx.md5("saltsalt" .. ngx.var.remote_addr .. "saltsalt" .. s)

ngx.header["Set-Cookie"] = {"coa=" .. coa .."; path=/; domain=.example.com; HttpOnly",

"cob=" .. s.."; path=/; domain=.example.com; HttpOnly"}

ngx.header["Content-Type"] = "text/html"

ngx.say("<meta http-equiv ='refresh' content='0'>");

ngx.exit(200)

end

}

# fastcgi_pass

}

其他

植入cookie

~~需要注意的是使用ngx.time()產生秒級的時間，用來做隨機數種子可能會衝突，因此建議加上另外的隨機變量（如下面的例子用的是客戶端的ip）~~ 可以使用ngx.now()產生毫秒精度時間

local modname = "util"
local _M = { _VERSION = '0.01' }
local mt = { __index = _M }

function _M.random_str(l, seed, r)
    local s = r or 'abcdefghijklmnhopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789=+-'
 
    local ret =''
	-- os.time() is precise to second
	math.randomseed(os.time() * 1000 + ngx.crc32_short(seed or ""))
    for i=1 ,l do
        local pos = math.random(1, string.len(s))
        ret = ret .. string.sub(s, pos, pos)
    end
 
    return ret
end

function _M.plant_cookie(k, h)
	local ck = require "resty.cookie"
	local cookie, err = ck:new()
	if not cookie then
		ngx.log(ngx.ERR, "ERROR PLANTING COOKIE", err)
		return
	end
	
	local _, err = cookie:get(k)
	if _ then
		return
	end
	local ok, err = cookie:set({
		key = k, value = _M.random_str(32, ngx.var.remote_addr, '0123456789abcdef'), path = "/",
		domain = h or ngx.var.http_host,
		httponly = true,
		expires= 'Thu, 31-Dec-37 23:55:55 GMT',
		max_age = 2147483647
	})

end

return _M

local modname = "util"

local _M = { _VERSION = '0.01' }

local mt = { __index = _M }

function _M.random_str(l, seed, r)

local s = r or 'abcdefghijklmnhopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789=+-'

local ret =''

-- os.time() is precise to second

math.randomseed(os.time() * 1000 + ngx.crc32_short(seed or ""))

for i=1 ,l do

local pos = math.random(1, string.len(s))

ret = ret .. string.sub(s, pos, pos)

end

return ret

end

function _M.plant_cookie(k, h)

local ck = require "resty.cookie"

local cookie, err = ck:new()

if not cookie then

ngx.log(ngx.ERR, "ERROR PLANTING COOKIE", err)

return

end

local _, err = cookie:get(k)

if _ then

return

end

local ok, err = cookie:set({

key = k, value = _M.random_str(32, ngx.var.remote_addr, '0123456789abcdef'), path = "/",

domain = h or ngx.var.http_host,

httponly = true,

expires= 'Thu, 31-Dec-37 23:55:55 GMT',

max_age = 2147483647

})

end

return _M

location / {
    access_by_lua '
        local _ = require("util")
        _.plant_cookie("cookie_name",".example.come")
    ';
}

location / {

access_by_lua '

local _ = require("util")

_.plant_cookie("cookie_name",".example.come")

}

Lua用setmetatable返回默認值為function時的暗坑

on 2015 年 5 月 30 日

0 119069 轉為簡體

想在api服務器里實現一個acl的功能，對某些請求（需要登錄，需要檢查appkey，需要限制頻次等）做限制，對某些起始狀態（比如登陸）或者終結狀態（比如報錯）的請求放行。

因為lua里木有switch case，因此通過一個acl_list的table去查找規則，因為需要限制的請求種類比較多，就把rule_check_token當成默認值了，一看是是這麼寫的：

local _M = {
    ACL_PASS = 0,
    ACL_DENIED = 1,
}

function _M.rule_always_pass(self)
    return _M.ACL_PASS
end

function _M.rule_check_token(query)
    if query == nil then
        return _M.ACL_DENIED
    end
    local token = query['token'] or ngx.req.get_headers()['Authorization']
    local uid = query['uid']
    -- ...
end

_M.acl_list = setmetatable({
    ['error'] = _M.rule_always_pass,
    ['user/login'] = _M.rule_always_pass,
}, {
    __index = _M.rule_check_token
})

return _M

local _M = {

ACL_PASS = 0,

ACL_DENIED = 1,

}

function _M.rule_always_pass(self)

return _M.ACL_PASS

end

function _M.rule_check_token(query)

if query == nil then

return _M.ACL_DENIED

end

local token = query['token'] or ngx.req.get_headers()['Authorization']

local uid = query['uid']

-- ...

end

_M.acl_list = setmetatable({

['error'] = _M.rule_always_pass,

['user/login'] = _M.rule_always_pass,

}, {

__index = _M.rule_check_token

})

return _M

然而卻會在local token= xxxx那一行報stack overflow，想了半天也發現哪裡有無限遞歸，因為query傳進來的是http請求的query string解析出的鍵值對錶。

把query打印出來一看，發現竟然是這個模塊本身……

仔細看了文檔才知道，原來__index後面的值是一個function時，lua會調用這個function去獲得不存在的鍵，並且第一個參數是模塊本身（即_M，一個table）。在這個例子里：

調用一個非默認規則的api
lua調用了rule_check_token
參數query被傳入了_M本身
運行到local token= xxxx這一行
這個table里(_M)又沒有token這個鍵
回到2

所以就死循環了

所以要好好看文檔

解決方法是可以套一個function

 __index = function() return _M.rule_check_token end

1	__index = function() return _M.rule_check_token end

貼一個打印table的工具，方便調試：

function print_r(tbl, pref)
    for k,v in pairs(tbl) do 
        if type(v) == 'table' then
            print((pref or "|-"), k, "\t", "<table>")
            _print_r(v, "  "..(pref or "|-"))
        else
            if type(v) == 'function' then
                v = "   <function>"
            end
            print((pref or "|-"), k, "\t", v)
        end
    end
end

function print_r(tbl, pref)

for k,v in pairs(tbl) do

if type(v) == 'table' then

print((pref or "|-"), k, "\t", "<table>")

_print_r(v, " "..(pref or "|-"))

else

if type(v) == 'function' then

v = " <function>"

end

print((pref or "|-"), k, "\t", v)

end

可以打印出如下形式：

|-_VERSION      0.01
|-rule_always_pass         <function>
|-ACL_DENIED    1
|-acl_list      <table>
  |-error          <function>
  |-user/login     <function>
|-ACL_PASS      0
|-rule_check_token         <function>

|-_VERSION 0.01

|-rule_always_pass <function>

|-ACL_DENIED 1

|-acl_list <table>

|-error <function>

|-user/login <function>

|-ACL_PASS 0

|-rule_check_token <function>

使用openresty請自行改寫成local function和ngx.say的形式

Category Archives

cidr.me地理位置查詢

使用方法

源代碼

用 OpenResty 寫了一個 SNI 代理

nginx 批量配置同步

使用shared dict

nginx/openresty的一些記錄

日誌

Header

防攻擊

其他

Lua用setmetatable返回默認值為function時的暗坑