2019年5月17日 星期五

電腦-將curl檢測url狀況及時間存成csv

電腦-將curl檢測url狀況及時間存成csv或json

電腦-使用curl檢測url狀況及時間中使用curl相關指令,對url進行一次性測試。
如果要長時間測試可以用script搭配cron排程進行。
script也將應用awk,curl中的相關時間進行計算如:
1.TCP連接時間(TCP Connection):%{time_pretransfer}-%{time_namelookup}
2.Server 處理時間(Content Generation、Processing Time):%{time_starttransfer}-%{time_pretransfer}
3.內容傳輸時間(Content Transfer):%{time_total}-%{time_starttransfer}



指定curl輸出格式
CURL_F='%{http_code},%{time_namelookup},%{time_connect},%{time_appconnect},%{time_pretransfer},%{time_redirect},%{time_starttransfer},%{time_total}'

curl指令
curl -w "$CURL_F" -o /dev/null -sS "$@"

#$@ 可代入的相關參數,如https://www.google.com.tw

指定awk輸出格式
AWK_F='$0"\nServer_Processing:"$7-$6"\nContent_Transfer:"$8-$7'

awk指令
awk 'BEGIN {FS=","}; {print '$DATE'",\n"'$AWK_F'}'


完整指令
==
DATE=`date +'%Y%m%d%H%M'`

CURL_F='%{http_code},%{time_namelookup},%{time_connect},%{time_appconnect},%{time_pretransfer},%{time_redirect},%{time_starttransfer},%{time_total}

AWK_F='$0","$7-$6","$8-$7'

curl -w "$CURL_F" -o /dev/null -sS "$@"| awk 'BEGIN {FS=","}; {print '$DATE'","'$AWK_F'}'

==

執行指令
./curltest.sh https://www.google.com.tw
輸出結果
201905171513,200,0.004,0.007,0.102,0.102,0.000,0.134,0.135,0.134,0.00

欄位分別對應:
時間,%{http_code},%{time_namelookup},%{time_connect},%{time_appconnect},%{time_pretransfer},%{time_redirect},%{time_starttransfer},%{time_total},Server Processing,Content Transfer




參考資料:
https://github.com/mat/dotfiles/blob/master/bin/curlt
Measuring HTTP response times with cURL
What Do Curl Timings Mean?
HTTP Request Timings with cURL
HTTP Transaction Timing Breakdown with cURL
Performance Testing with cURL, Part 1: Basics
Performance Testing with cURL, Part 2: Scripting

2019年5月16日 星期四

電腦-使用curl檢測url狀況及時間

電腦-使用curl檢測url狀況及時間

為了檢測url回應速度,可以使用curl指令,提供連線時各階段的時間,方便分析。
測試主要使用在Centos、Ubuntu。

主要會用到的參數
-w, --write-out <format> 本次主要以時間和url為主,各項時間說明可參考後續。
-o  /dev/null:把回應內容丟棄 。(windows 請使用null)
-s,--silent
-S,--show-error
-L,果服務器報告所請求的頁面已移動到其他位置,則此選項將使curl重做新位置的請求,將有2段時間。

指令
curl -sSo /dev/null -w " Time_namelookup:%{time_namelookup}\n TCP_Connect time:%{time_connect}\n  SSL_connect time:%{time_appconnect}\n Redirect time:%{time_redirect}\n Pretransfer:%{time_pretransfer}\n Starttransfer:%{time_starttransfer}\n Time_total:%{time_total}\n speed_download:%{speed_download}\n" https://www.google.com

輸出結果
 Time_namelookup:0.004                                             
 TCP_Connect time:0.015
 SSL_connect time:0.141
 Redirect time:0.000
 Pretransfer:0.141
 Starttransfer:0.183
 time_total:0.186
 speed_download:63933.000

 時間單位皆為秒。

 Time_namelookup %{time_namelookup}
 DNS 解析域名[www.google.com]的時間

TCP_Connect time:%{time_connect}
 client和server端建立TCP 連接的時間

 SSL_connect time:%{time_appconnect}
 SSL/SSH 等上層協議建立連接的時間,比如 connect/handshake 的時間

 Redirect time:%{time_redirect}
 所有重定向步驟(包括名稱查找,連接,預傳輸和傳輸)所花費的時間

 Pretransfer:%{time_pretransfer}
 從請求開始到響應開始傳輸的時

 Starttransfer:%{time_starttransfer}
 從client發出請求到web的server 響應第一個字節的時間

 Time_total:%{time_total}
 client發出請求;到web的server發送會所有的相應數據的時間

 speed_download:%{speed_download}
 下載速度 單位 byte/s

 根據以上時間,可以計算出以下時間
 1.TCP連接時間(TCP Connection):%{time_pretransfer}-%{time_namelookup}
 2.Server 處理時間(Content Generation、Processing Time):%{time_starttransfer}-%{time_pretransfer}
 3.內容傳輸時間(Content Transfer):%{time_total}-%{time_starttransfer}


如果指令太長,可以把先將curl輸出內容存成檔案(curl-format.txt)
\n
Time_namelookup:  %{time_namelookup}\n
Time_connect:  %{time_connect}\n
Time_appconnect:  %{time_appconnect}\n
Time_redirect:  %{time_redirect}\n
Time_pretransfer:  %{time_pretransfer}\n
Time_starttransfer:  %{time_starttransfer}\n
Time_total:  %{time_total}\n
\n

指令
curl -w @./curl-format.txt -sSo /dev/null https://www.google.com.tw

輸出結果
Time_namelookup:  0.004
Time_connect:  0.013
Time_appconnect:  0.127
Time_redirect:  0.000
Time_pretransfer:  0.127
Time_starttransfer:  0.179
----------
Time_total:  0.182


curl 各項時間參數
time_appconnect
The time, in seconds, it took from the start until the SSL/SSH/etc connect/handshake to the remote host was completed. (Added in 7.19.0)

time_connect
The time, in seconds, it took from the start until the TCP connect to the remote host (or proxy) was completed.

time_namelookup
The time, in seconds, it took from the start until the name resolving was completed.

time_pretransfer
The time, in seconds, it took from the start until the file transfer was just about to begin. This includes all pre-transfer commands and negotiations that are specific to the particular protocol(s) involved.

time_redirect
The time, in seconds, it took for all redirection steps including name lookup, connect, pretransfer and transfer before the final transaction was started. time_redirect shows the complete execution time for multiple redirections. (Added in 7.12.3)

time_starttransfer
The time, in seconds, it took from the start until the first byte was just about to be transferred. This includes time_pretransfer and also the time the server needed to calculate the result.

time_total
The total time, in seconds, that the full operation lasted.

url_effective
The URL that was fetched last. This is most meaningful if you've told curl to follow location: headers.


參考資料:
curl.1 the man page
使用 curl 命令分析请求的耗时情况
CURL 於 CLI 顯示 TCP、SSL 的連線時間
Performance Testing with cURL, Part 1: Basics
Curl 查看網頁載入前的各項時間 (windows)

2018年12月4日 星期二

電腦-Linux-CentOS 服務設定

電腦-Linux-CentOS 服務設定

CentOS7
服務管理使用systemd
服務啟動檔位置 /usr/lib/systemd/system/
服務設定為enable,將產生link檔至/etc/systemd/system中
啟動
systemctl start service
停止
systemctl stop service
開機啟用
systemctl enable service
開機停用
systemctl disable service
檢查服務狀態
systemctl status service
重啟服務
systemctl restart service

列出所有服務
systemctl list-units-files
列出已啟動的服務
systemctl list-units --type service | grep running


CentOS6
服務管理使用System V
服務啟動檔位置/etc/init.d/
啟動
server service start
停止
server service stop
檢查服務狀態
server service status
重啟服務
server service restart
開機啟用
chkconifg service on
開機停用
chkconfig service off
列出所有服務
chkconfig --list
新增服務
chkconfig -add service
刪除服務
chkconfig -del service



參考資料

鳥哥的Linux 私房菜-- 第十七章、認識系統服務(daemons)

2018年11月29日 星期四

電腦-Linux-Ubuntu 安裝 Docker CE

電腦-Linux-Ubuntu安裝Docker CE

Ubuntu 16.04
Docker CE (18.x)

移除舊版本
apt-get remove docker docker-engine docker.io containerd runc

安裝相關套件
sudo apt-get update
sudo apt-get install apt-transport-https ca-certificates curl software-properties-common

安裝Docker GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

安裝Docker Repository
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"

安裝Docker-ce
sudo apt-get update
sudo apt-get install docker-ce

檢查Docker服務
sudo systemctl status docker

檢查Docker服務
sudo docker -v
apt-cache madison docker-ce

指定安裝版本
sudo apt-get install docker-ce=<VERSION>

新增一般使用者至dokcer group,指令可不加sudo
sudo usermod -aG docker $(whoami)
檢查使用者group
id -nG

移除Docker CE
sudo apt-get purge docker-ce
移除Docker相關資料
sudo rm -rf /var/lib/docker



參考資料
Get Docker CE for Ubuntu
How To Install and Use Docker on Ubuntu 16.04

2018年11月28日 星期三

電腦-Linux-Librenms 安裝(Centos+nginx)

電腦-Linux-Librenms 安裝(Centos+Nginx)

CentOS7 安裝 LibreNMS

安裝環境
CentOS 7
DB使用 MariaDB/MySQL Galera Cluster + Haproxy

安裝套件(epel、webtatic、php72)
yum install epel-release
rpm -Uvh https://mirror.webtatic.com/yum/el7/webtatic-release.rpm
yum install composer cronie fping git ImageMagick jwhois mtr MySQL-python net-snmp net-snmp-utils nginx nmap php72w php72w-cli php72w-common php72w-curl php72w-fpm php72w-gd php72w-mbstring php72w-mysqlnd php72w-process php72w-snmp php72w-xml php72w-zip python-memcached rrdtool

建立 librenms 用戶,加入 nginx 群組
useradd librenms -d /opt/librenms -M -r
usermod -a -G librenms nginx

下載 LibreNMS
cd /opt
composer create-project --no-dev --keep-vcs librenms/librenms librenms dev-master

設定 DB Server
登入MariaDB/MySQL Galera Cluster 第一台主機,建立librenms資料庫。

CREATE DATABASE librenms CHARACTER SET utf8 COLLATE utf8_unicode_ci;
CREATE USER 'librenms'@'localhost' IDENTIFIED BY 'librenms';
GRANT ALL PRIVILEGES ON librenms.* TO 'librenms'@'localhost';
FLUSH PRIVILEGES;
exit

登入MariaDB/MySQL Galera Cluster 所有主機,修改設定檔。
vim /etc/my.cnf.d/server.cnf
==
[mysqld]
innodb_file_per_table=1
sql-mode=""
lower_case_table_names=0
==
systemctl enable mariadb
systemctl restart mariadb

建立 Web Server (PHP-FPM+Nginx)

設定PHP
vim  /etc/php.ini
==
[Date]
date.timezone = Asia/Taipei
==

設定 PHP-FPM
vim /etc/php-fpm.d/www.conf
==
user = nginx
group = apache

listen = /var/run/php-fpm/php7.2-fpm.sock

listen.owner = nginx
listen.group = nginx
listen.mode = 0660
==
systemctl enable php-fpm
systemctl restart php-fpm

設定 nginx

停用nginx default 設定
vi /etc/nginx/nginx.conf
==
    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

#    server {
#        listen       80 default_server;
#        listen       [::]:80 default_server;
#        server_name  _;
#        root         /usr/share/nginx/html;
#
#        # Load configuration files for the default server block.
#        include /etc/nginx/default.d/*.conf;
#
#        location / {
#        }
#
#        error_page 404 /404.html;
#            location = /40x.html {
#        }
#
#        error_page 500 502 503 504 /50x.html;
#            location = /50x.html {
#        }
#    }

# Settings for a TLS enabled server.
#
#    server {
==

新增librenms 設定檔
vim /etc/nginx/conf.d/librenms.conf
==
server {
 listen      80;
 server_name 192.168.11.201;
 root        /opt/librenms/html;
 index       index.php;

 charset utf-8;
 gzip on;
 gzip_types text/css application/javascript text/javascript application/x-javascript image/svg+xml text/plain text/xsd text/xsl text/xml image/x-icon;
 location / {
  try_files $uri $uri/ /index.php?$query_string;
 }
 location /api/v0 {
  try_files $uri $uri/ /api_v0.php?$query_string;
 }
 location ~ \.php {
  include fastcgi.conf;
  fastcgi_split_path_info ^(.+\.php)(/.+)$;
  fastcgi_pass unix:/var/run/php-fpm/php7.2-fpm.sock;
 }
 location ~ /\.ht {
  deny all;
 }
}
==
systemctl enable nginx
systemctl restart nginx

修改 SELinux 與 Allow fping
安裝SELinux policy tool
yum install policycoreutils-python

設定權限
semanage fcontext -a -t httpd_sys_content_t '/opt/librenms/logs(/.*)?'
semanage fcontext -a -t httpd_sys_rw_content_t '/opt/librenms/logs(/.*)?'
restorecon -RFvv /opt/librenms/logs/
semanage fcontext -a -t httpd_sys_content_t '/opt/librenms/rrd(/.*)?'
semanage fcontext -a -t httpd_sys_rw_content_t '/opt/librenms/rrd(/.*)?'
restorecon -RFvv /opt/librenms/rrd/
semanage fcontext -a -t httpd_sys_content_t '/opt/librenms/storage(/.*)?'
semanage fcontext -a -t httpd_sys_rw_content_t '/opt/librenms/storage(/.*)?'
restorecon -RFvv /opt/librenms/storage/
semanage fcontext -a -t httpd_sys_content_t '/opt/librenms/bootstrap/cache(/.*)?'
semanage fcontext -a -t httpd_sys_rw_content_t '/opt/librenms/bootstrap/cache(/.*)?'
restorecon -RFvv /opt/librenms/bootstrap/cache/
setsebool -P httpd_can_sendmail=1
setsebool -P httpd_execmem 1

新增http_fping.tt
vim /tmp/http_fping.tt
==
module http_fping 1.0;

require {
type httpd_t;
class capability net_raw;
class rawip_socket { getopt create setopt write read };
}

#============= httpd_t ==============
allow httpd_t self:capability net_raw;
allow httpd_t self:rawip_socket { getopt create setopt write read };
==
checkmodule -M -m -o http_fping.mod /tmp/http_fping.tt
semodule_package -o /tmp/http_fping.pp -m http_fping.mod
semodule -i /tmp/http_fping.pp


設定 firewall
firewall-cmd --zone public --add-service http
firewall-cmd --permanent --zone public --add-service http
firewall-cmd --zone public --add-service https
firewall-cmd --permanent --zone public --add-service https


設定 snmpd
cp /opt/librenms/snmpd.conf.example /etc/snmp/snmpd.conf

vim /etc/snmp/snmpd.conf
==
將 RANDOMSTRINGGOESHERE 改成你要的 community 字串
com2sec readonly  default         public

更改位置及個人資料
syslocation Taipei, Taiwan
syscontact  Your Name <your@email.address>
==
curl -o /usr/bin/distro https://raw.githubusercontent.com/librenms/librenms-agent/master/snmp/distro
chmod +x /usr/bin/distro

systemctl enable snmpd
systemctl restart snmpd


設定 cron table
cp /opt/librenms/librenms.nonroot.cron /etc/cron.d/librenms

設定定時清除 log
cp /opt/librenms/misc/librenms.logrotate /etc/logrotate.d/librenms

設定權限
chown -R librenms:librenms /opt/librenms
setfacl -d -m g::rwx /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
setfacl -R -m g::rwx /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/

網頁安裝 http://192.168.11.201/install.php

依照網頁出現的 6 個步驟執行
步驟5將產生的設定參數,新增/opt/librenms/config.php 手動寫入
vim /opt/librenms/config.php

調整參數檔案權限
chown librenms:librenms /opt/librenms/config.php

再來應該就可以透過網頁依照上一步驟建立的帳號登入


參考資料
原廠 Installation-CentOS-7-Nginx
Librenms 安裝於 Centos7
CentOS7 安裝 LibreNMS
在 CentOS 7 / RHEL 7上,安裝 LibreNMS 服務

電腦-Linux-Haproxy 設定檔(MariaDB/MySQL)

電腦-Linux-Haproxy 設定檔(MariaDB/MySQL)

配合之前建立 MariaDB/MySQL 安裝 Galera Cluster ,需要使用Haproxy來做負載平衡。

global
    log         127.0.0.1 local0
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

defaults
    mode                            http
    log                                global
    option                           httplog
    option                           dontlognull
    retries                           3
    timeout http-request      10s
    timeout queue               1m
    timeout connect            10s
    timeout client                1m
    timeout server               1m
    timeout http-keep-alive 10s
    timeout check               10s
    maxconn                       500

#管理頁面
listen stats
    bind 0.0.0.0:8181
    stats enable
    stats hide-version
    stats uri /admin
    stats auth admin:admin
    stats admin if TRUE
    stats refresh 30s

#MariaDB/MySQL
frontend MySQL
    bind *:3306
    mode tcp
    option tcplog
    capture request header Host len 64
    capture request header User-Agent len 128
    capture request header X-Forwarded-For len 100
    capture request header Referer len 200
    capture response header Server len 40
    capture response header Server-ID len 40
    log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq"

default_backend MySQL
    mode tcp
    balance leastconn
    server db01 192.168.11.101:3306
    server db02 192.168.11.102:3306


參考資料
Percona XtraDB Cluster 搭配 HAProxy

電腦-Linux-Haproxy 設定檔(http/https)

電腦-Linux-Haproxy 設定檔(http/https)

Haproxy安裝ssl憑證,後端server使用http。
當使用者使用http會自動轉為https,根據url指向定義的server。

global
    log         127.0.0.1 local0
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    tune.ssl.default-dh-param 2048
    ssl-default-bind-ciphers DES:RC4:AES256-SHA:AES128-SHA:DES-CBC3-SHA:!ADH-RC4-MD5:!ECDHE-RSA-RC4-SHA:!AECDH-RC4-SHA

defaults
    mode                            http
    log                                global
    option                           httplog
    option                           dontlognull
    retries                           3
    timeout http-request      10s
    timeout queue               1m
    timeout connect            10s
    timeout client                1m
    timeout server               1m
    timeout http-keep-alive 10s
    timeout check               10s
    maxconn                       3000

#管理頁面
listen stats
    bind 0.0.0.0:8181
    stats enable
    stats hide-version
    stats uri /admin
    stats auth admin:admin
    stats admin if TRUE
    stats refresh 30s

#http轉https
frontend  http
bind *:80
redirect scheme https code 301 if !{ ssl_fc }

#https
frontend https_switch
    bind *:443 ssl crt /etc/pki/CA/CA.pem
    mode http
    option httpclose
    option forwardfor
    reqadd X-Forwarded-Proto:\ https

#ACL
    acl url_testsite ssl_fc_sni testsite.com
    use_backend url_testsite if url_testsite
    acl url_portal ssl_fc_sni -i portal.com
    use_backend url_portal if url_portal

#log-format
    capture request header Host len 64
    capture request header User-Agent len 128
    capture request header X-Forwarded-For len 100
    capture request header Referer len 200
    capture response header Server len 40
    capture response header Server-ID len 40
    log-format %ci:%cp\ %si:%sp\ %B\ %U\ %ST\ %r\ %b\ %f\ %bi\ %hrl\ %hsl\


backend url_testsite
    mode    http
    balance roundrobin
    option  httpclose
    option  forwardfor
    cookie  SRVERID insert nocache
    server  testserver1 192.168.11.11:80 check inter 3000 rise 3 fall 5
    server  testserver2 192.168.11.12:80 check inter 3000 rise 3 fall 5 backup
    #backup為備援參數,當testserver1異常才會使用

backend url_portal
    mode    http
    balance roundrobin
    option  httpclose
    option  forwardfor
    cookie  SRVERID insert nocache
    server  webserver1 192.168.11.21:80 check inter 3000 rise 3 fall 5
    server  webserver2 192.168.11.22:80 check inter 3000 rise 3 fall 5 backup


參考資料
haproxy生产环境的一个完整配置