Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: run test cases of APISIX on Tengine + lua-nginx-module #677

Closed
membphis opened this issue Oct 12, 2019 · 1 comment · Fixed by #683
Closed

test: run test cases of APISIX on Tengine + lua-nginx-module #677

membphis opened this issue Oct 12, 2019 · 1 comment · Fixed by #683
Labels

Comments

@membphis
Copy link
Member

membphis commented Oct 12, 2019

非常感谢 @totemofwolf 的安装步骤信息。

Tengine + openresty, here are the compiling steps:

  • 之前一直使用tengine,上个正式版还是2015年...之后一年更新一个开发版本,望眼欲穿
  • 2019年发版本频率明显提升(可能是合并到主线了?)
  • 目前在k8s里的流量网关也是使用tengine,主要用 ngx_http_upstream_dynamic_module 做服务发现
  • 看到各种基于OpenResty的魔改软件横空出世,于是想着用tengine + ngx_lua_module,发现他俩合起来仍然不是OpenResty
  • 偶然看到别人文章可以替换 openresty/bundle的nginx,于是有了此文
  • 本次主要替换/升级了如下依赖:
    • tengine-2.3.2
    • luajit2-2.1-20190912
    • ngx_devel_kit-0.3.1
    • redis-nginx-module-0.3.9
#
➜ docker run --rm -it alpine:3.7

# 以下操作步骤均在此docker中进行
/bin/sed -i 's,http://dl-cdn.alpinelinux.org,https://mirrors.aliyun.com,g' /etc/apk/repositories
# apk update
apk add --no-cache curl make zlib-dev pcre-dev openssl-dev libc-dev gcc jemalloc-dev g++ geoip-dev tzdata perl linux-headers git

1. 获取依赖包:

mkdir -pv /tmp/apisix
cd /tmp/apisix
# 获取 openresty-1.15.8.2
curl -sk https://openresty.org/download/openresty-1.15.8.2.tar.gz | tar zxf - -C .

# 获取 tengine-2.3.2
curl -sk https://tengine.taobao.org/download/tengine-2.3.2.tar.gz | tar zxf - -C .

# 获取 luajit2-2.1-20190912
curl -sk https://codeload.github.com/openresty/luajit2/tar.gz/v2.1-20190912 | tar zxf - -C .

# 获取 ngx_devel_kit-0.3.1
curl -sk https://codeload.github.com/simplresty/ngx_devel_kit/tar.gz/v0.3.1 | tar zxf - -C .

# 获取 redis-nginx-module-0.3.9
# https://www.nginx.com/resources/wiki/modules/redis/
curl -sk https://people.freebsd.org/~osa/ngx_http_redis-0.3.9.tar.gz | tar zxf - -C .
mv ngx_http_redis-0.3.9 redis-nginx-module-0.3.9

2. 魔改:

#
rm -rf openresty-1.15.8.2/bundle/nginx-1.15.8
mv tengine-2.3.2 openresty-1.15.8.2/bundle/

#
rm -rf openresty-1.15.8.2/bundle/LuaJIT-2.1-20190507
mv luajit2-2.1-20190912 openresty-1.15.8.2/bundle/

#
rm -rf openresty-1.15.8.2/bundle/ngx_devel_kit-0.3.1rc1
mv ngx_devel_kit-0.3.1 openresty-1.15.8.2/bundle/

#
rm -rf openresty-1.15.8.2/bundle/redis-nginx-module-0.3.7
mv redis-nginx-module-0.3.9 openresty-1.15.8.2/bundle/

3. 修改openresty-1.15.8.2/configure文件:

# colordiff configure configure.old
689c689
<         my $luajit_src = auto_complete 'luajit2';
---
>         my $luajit_src = auto_complete 'LuaJIT';
1243c1243
<     $ngx_dir = auto_complete "tengine";
---
>     $ngx_dir = auto_complete "nginx";

4. 编译安装:

特别地,如果你的宿主机是Mac,你可以将上面的3步骤在宿主机上执行,然后进行以下操作拷贝包到容器:
# find . -maxdepth 7 -type f -name '.DS_Store' | xargs rm -fv
# tar zcf openresty-1.15.8.2-lt.tar.gz openresty-1.15.8.2
# sudo httpd -t
# sudo httpd -k start
# cp openresty-1.15.8.2-lt.tar.gz /Library/WebServer/Documents/
# 然后直接在容器里执行
# curl -s http://mossdeMBP2/openresty-1.15.8.2-lt.tar.gz | tar zxf - -C /usr/local/src/
# cd /usr/local/src/openresty-*/
# mossdeMBP2 来自宿主机Mac的 `hostname` 输出
./configure -j4 \
--sbin-path=/usr/sbin/nginx \
--modules-path=/usr/lib64/nginx/modules \
--conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--pid-path=/var/run/nginx.pid \
--lock-path=/var/run/nginx.lock \
--http-client-body-temp-path=/var/cache/nginx/client_temp \
--http-proxy-temp-path=/var/cache/nginx/proxy_temp \
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \
--http-scgi-temp-path=/var/cache/nginx/scgi_temp \
--user=nginx \
--group=nginx \
--with-compat \
--with-file-aio \
--with-threads \
--with-http_addition_module \
--with-http_auth_request_module \
--with-http_dav_module \
--with-http_degradation_module \
--with-http_flv_module \
--with-http_geoip_module=dynamic \
--with-http_gunzip_module \
--with-http_gzip_static_module \
--with-http_mp4_module \
--with-http_random_index_module \
--with-http_realip_module \
--with-http_secure_link_module \
--with-http_ssl_module \
--with-http_stub_status_module \
--with-http_sub_module \
--with-http_v2_module \
--with-stream \
--with-stream_ssl_module \
--with-stream_realip_module \
--with-stream_geoip_module=dynamic \
--with-stream_ssl_preread_module \
--with-stream_sni \
--with-jemalloc \
--with-pcre \
--with-pcre-jit \
--without-mail_pop3_module \
--without-mail_imap_module \
--without-mail_smtp_module \
--add-module=bundle/tengine-2.3.2/modules/ngx_http_upstream_vnswrr_module/ \
--add-module=bundle/tengine-2.3.2/modules/mod_dubbo \
--add-module=bundle/tengine-2.3.2/modules/ngx_multi_upstream_module \
--add-module=bundle/tengine-2.3.2/modules/mod_config \
--add-dynamic-module=bundle/tengine-2.3.2/modules/ngx_http_concat_module \
--add-dynamic-module=bundle/tengine-2.3.2/modules/ngx_http_footer_filter_module \
--add-dynamic-module=bundle/tengine-2.3.2/modules/ngx_http_proxy_connect_module \
--add-dynamic-module=bundle/tengine-2.3.2/modules/ngx_http_reqstat_module \
--add-dynamic-module=bundle/tengine-2.3.2/modules/ngx_http_slice_module \
--add-dynamic-module=bundle/tengine-2.3.2/modules/ngx_http_sysguard_module \
--add-dynamic-module=bundle/tengine-2.3.2/modules/ngx_http_trim_filter_module \
--add-dynamic-module=bundle/tengine-2.3.2/modules/ngx_http_upstream_check_module \
--add-dynamic-module=bundle/tengine-2.3.2/modules/ngx_http_upstream_consistent_hash_module \
--add-dynamic-module=bundle/tengine-2.3.2/modules/ngx_http_upstream_dynamic_module \
--add-dynamic-module=bundle/tengine-2.3.2/modules/ngx_http_upstream_dyups_module \
--add-dynamic-module=bundle/tengine-2.3.2/modules/ngx_http_upstream_session_sticky_module \
--add-dynamic-module=bundle/tengine-2.3.2/modules/ngx_http_user_agent_module \
--add-dynamic-module=bundle/tengine-2.3.2/modules/ngx_slab_stat

# 不出意外出现:
# checking for PCRE library ... found
# checking for PCRE JIT support ... found
# checking for OpenSSL library ... found
# checking for zlib library ... found
# checking for GeoIP library ... found
# checking for GeoIP IPv6 support ... found
# checking for jemalloc library ... found
# creating objs/Makefile

# Configuration summary
#   + using threads
#   + using system PCRE library
#   + using system OpenSSL library
#   + using system zlib library
#   + using system jemalloc library

#   nginx path prefix: "/usr/local/openresty/nginx"
#   nginx binary file: "/usr/sbin/nginx"
#   nginx modules path: "/usr/lib64/nginx/modules"
#   nginx configuration prefix: "/etc/nginx"
#   nginx configuration file: "/etc/nginx/nginx.conf"
#   nginx pid file: "/var/run/nginx.pid"
#   nginx error log file: "/var/log/nginx/error.log"
#   nginx http access log file: "/var/log/nginx/access.log"
#   nginx http client request body temporary files: "/var/cache/nginx/client_temp"
#   nginx http proxy temporary files: "/var/cache/nginx/proxy_temp"
#   nginx http fastcgi temporary files: "/var/cache/nginx/fastcgi_temp"
#   nginx http uwsgi temporary files: "/var/cache/nginx/uwsgi_temp"
#   nginx http scgi temporary files: "/var/cache/nginx/scgi_temp"

# cd ../..
# Type the following commands to build and install:
#     make
#     make install

# build
make -j4
make install clean

mkdir -pv /var/cache/nginx/client_temp
mkdir -pv /etc/nginx/conf.d

adduser nginx -s /sbin/nologin -D

5. 配置文件

# > /etc/nginx/nginx.conf

# daemon off;
# or use 'CMD ["nginx", "-g", "daemon off;"]' in Dockerfile
#user  nobody;

worker_processes  auto;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;
#error_log  "pipe:rollback logs/error_log interval=1d baknum=7 maxsize=2G";

#pid        logs/nginx.pid;

load_module "/usr/lib64/nginx/modules/ngx_http_concat_module.so";
load_module "/usr/lib64/nginx/modules/ngx_http_dyups_module.so";
load_module "/usr/lib64/nginx/modules/ngx_http_footer_filter_module.so";
load_module "/usr/lib64/nginx/modules/ngx_http_geoip_module.so";
load_module "/usr/lib64/nginx/modules/ngx_http_proxy_connect_module.so";
load_module "/usr/lib64/nginx/modules/ngx_http_reqstat_module.so";
load_module "/usr/lib64/nginx/modules/ngx_http_trim_filter_module.so";
load_module "/usr/lib64/nginx/modules/ngx_http_upstream_dynamic_module.so";
load_module "/usr/lib64/nginx/modules/ngx_http_user_agent_module.so";
load_module "/usr/lib64/nginx/modules/ngx_stream_geoip_module.so";

events {
    worker_connections  10240;
    use epoll;
}

# load modules compiled as Dynamic Shared Object (DSO)
#
#dso {
#    load ngx_http_fastcgi_module.so;
#    load ngx_http_rewrite_module.so;
#}

http {
    lua_code_cache on;
    # lua_code_cache off;
    # lua_package_path "lua/?.lua;lua/lib/?.lua;lua/lib/lua-resty-core/lib/?.lua;;";
    # lua_package_path "/usr/local/lib/lua/?.lua;;";
    lua_need_request_body on;

    include       mime.types;
    default_type  application/octet-stream;

    log_format  mains '$remote_addr | $status | "$request"'
    #'$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;
    #access_log  "pipe:rollback logs/access_log interval=1d baknum=7 maxsize=2G"  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

# tricks
    server_tokens off;
    #Server头伪装
    server_tag 'Apache';
    server_info off;
    proxy_hide_header "X-Powered-By";
    proxy_hide_header "Server";

# security : for version <= 2.1.2
    #CVE-2017-7529
    #max_ranges 1;

# Buffers
# https://www.mtyun.com/library/how-to-optimize-nginx
    client_body_buffer_size 10k;
    #client_max_body_size 100m;
    # nginx: [warn] client_max_body_size 51200 should be greater than total postpone buffer size 65536 in /etc/nginx/nginx.conf:169
    #client_max_body_size 0;
    client_header_buffer_size 4k;
    large_client_header_buffers 2 4k;

    proxy_buffer_size 512k;
    proxy_buffers 8 512k;
    proxy_busy_buffers_size 512k;
    proxy_temp_file_write_size 512k;

    proxy_headers_hash_max_size 1024;
    proxy_headers_hash_bucket_size 128;
# Buffers end

# Timeouts
# proxy_pass timeout
    proxy_connect_timeout       5;
    proxy_send_timeout          300;
    proxy_read_timeout          300;

    client_body_timeout 12;
    client_header_timeout 12;
    # keepalive_timeout 15;
    send_timeout 10;
# Timeouts end

    include conf.d/*.conf;
}
# > /etc/nginx/conf.d/gee.conf
server {
    listen 80;
    server_name  gee.nsa.info *.nsa.info;

    # allow large uploads of files - refer to nginx documentation
    # client_max_body_size 1G;

    # optimize downloading files larger than 1G - refer to nginx doc before adjusting
    # proxy_max_temp_file_size 2G;

    # # [Optional] Enable HTTP Strict Transport Security
    # # HSTS is a feature improving protection against MITM attacks
    # # For more information see: https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/
    #add_header Strict-Transport-Security "max-age=31536000";

    #auth_basic "Protected Elasticsearch";
    #auth_basic_user_file es_passwords;

    location /lua {
        #access_by_lua_file 'authorize.lua';
        default_type    'text/html';
        content_by_lua    "ngx.say('Hello  World')";
    }

     location /nginx_var {
         # MIME type determined by default_type:
         default_type 'text/plain';

         # try access /nginx_var?a=hello,world
         content_by_lua_block {
             ngx.say(ngx.var.arg_a)
         }
     }

     location = /request_body {
	 # nginx: [warn] client_max_body_size 51200 should be greater than total postpone buffer size 65536 in /etc/nginx/nginx.conf:170
         # client_max_body_size 50k;
         client_body_buffer_size 50k;

         content_by_lua_block {
             ngx.req.read_body()  -- explicitly read the req body
             local data = ngx.req.get_body_data()
             if data then
                 ngx.say("body data:")
                 ngx.print(data)
                 return
             end

             -- body may get buffered in a temp file:
             local file = ngx.req.get_body_file()
             if file then
                 ngx.say("body is in file ", file)
             else
                 ngx.say("no body found")
             end
         }
     }
}

6. 测试运行状况:

echo "export PATH=$PATH:/usr/local/openresty/bin" >> /etc/profile
source /etc/profile

nginx -v
# Tengine version: Tengine/2.3.2
# nginx version: nginx/1.17.3

nginx -t
# nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
# nginx: configuration file /etc/nginx/nginx.conf test is successful
# or
openresty -t
# nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
# nginx: configuration file /etc/nginx/nginx.conf test is successful

# start
nginx

curl gee.nsa.info/lua -x 127.0.0.1:80
# Hello  World

curl gee.nsa.info/nginx_var?a=hello,world -x 127.0.0.1:80
# hello,world

# 更高级地测试
# cd ..
# git clone https://github.com/totemofwolf/lua-resty-redis-ratelimit.git
# cd lua-resty-redis-ratelimit/
# make install DESTDIR= LUA_LIB_DIR=/usr/local/openresty/lualib
# ...
@membphis
Copy link
Member Author

membphis commented Nov 2, 2019

We need to patch the Nginx core, the OpenResty core need it. We need to find a fully patch list.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant