2019年5月17日 星期五

電腦-將curl檢測url狀況及時間存成csv

電腦-將curl檢測url狀況及時間存成csv或json

電腦-使用curl檢測url狀況及時間中使用curl相關指令,對url進行一次性測試。
如果要長時間測試可以用script搭配cron排程進行。
script也將應用awk,curl中的相關時間進行計算如:
1.TCP連接時間(TCP Connection):%{time_pretransfer}-%{time_namelookup}
2.Server 處理時間(Content Generation、Processing Time):%{time_starttransfer}-%{time_pretransfer}
3.內容傳輸時間(Content Transfer):%{time_total}-%{time_starttransfer}



指定curl輸出格式
CURL_F='%{http_code},%{time_namelookup},%{time_connect},%{time_appconnect},%{time_pretransfer},%{time_redirect},%{time_starttransfer},%{time_total}'

curl指令
curl -w "$CURL_F" -o /dev/null -sS "$@"

#$@ 可代入的相關參數,如https://www.google.com.tw

指定awk輸出格式
AWK_F='$0"\nServer_Processing:"$7-$6"\nContent_Transfer:"$8-$7'

awk指令
awk 'BEGIN {FS=","}; {print '$DATE'",\n"'$AWK_F'}'


完整指令
==
DATE=`date +'%Y%m%d%H%M'`

CURL_F='%{http_code},%{time_namelookup},%{time_connect},%{time_appconnect},%{time_pretransfer},%{time_redirect},%{time_starttransfer},%{time_total}

AWK_F='$0","$7-$6","$8-$7'

curl -w "$CURL_F" -o /dev/null -sS "$@"| awk 'BEGIN {FS=","}; {print '$DATE'","'$AWK_F'}'

==

執行指令
./curltest.sh https://www.google.com.tw
輸出結果
201905171513,200,0.004,0.007,0.102,0.102,0.000,0.134,0.135,0.134,0.00

欄位分別對應:
時間,%{http_code},%{time_namelookup},%{time_connect},%{time_appconnect},%{time_pretransfer},%{time_redirect},%{time_starttransfer},%{time_total},Server Processing,Content Transfer




參考資料:
https://github.com/mat/dotfiles/blob/master/bin/curlt
Measuring HTTP response times with cURL
What Do Curl Timings Mean?
HTTP Request Timings with cURL
HTTP Transaction Timing Breakdown with cURL
Performance Testing with cURL, Part 1: Basics
Performance Testing with cURL, Part 2: Scripting

2019年5月16日 星期四

電腦-使用curl檢測url狀況及時間

電腦-使用curl檢測url狀況及時間

為了檢測url回應速度,可以使用curl指令,提供連線時各階段的時間,方便分析。
測試主要使用在Centos、Ubuntu。

主要會用到的參數
-w, --write-out <format> 本次主要以時間和url為主,各項時間說明可參考後續。
-o  /dev/null:把回應內容丟棄 。(windows 請使用null)
-s,--silent
-S,--show-error
-L,果服務器報告所請求的頁面已移動到其他位置,則此選項將使curl重做新位置的請求,將有2段時間。

指令
curl -sSo /dev/null -w " Time_namelookup:%{time_namelookup}\n TCP_Connect time:%{time_connect}\n  SSL_connect time:%{time_appconnect}\n Redirect time:%{time_redirect}\n Pretransfer:%{time_pretransfer}\n Starttransfer:%{time_starttransfer}\n Time_total:%{time_total}\n speed_download:%{speed_download}\n" https://www.google.com

輸出結果
 Time_namelookup:0.004                                             
 TCP_Connect time:0.015
 SSL_connect time:0.141
 Redirect time:0.000
 Pretransfer:0.141
 Starttransfer:0.183
 time_total:0.186
 speed_download:63933.000

 時間單位皆為秒。

 Time_namelookup %{time_namelookup}
 DNS 解析域名[www.google.com]的時間

TCP_Connect time:%{time_connect}
 client和server端建立TCP 連接的時間

 SSL_connect time:%{time_appconnect}
 SSL/SSH 等上層協議建立連接的時間,比如 connect/handshake 的時間

 Redirect time:%{time_redirect}
 所有重定向步驟(包括名稱查找,連接,預傳輸和傳輸)所花費的時間

 Pretransfer:%{time_pretransfer}
 從請求開始到響應開始傳輸的時

 Starttransfer:%{time_starttransfer}
 從client發出請求到web的server 響應第一個字節的時間

 Time_total:%{time_total}
 client發出請求;到web的server發送會所有的相應數據的時間

 speed_download:%{speed_download}
 下載速度 單位 byte/s

 根據以上時間,可以計算出以下時間
 1.TCP連接時間(TCP Connection):%{time_pretransfer}-%{time_namelookup}
 2.Server 處理時間(Content Generation、Processing Time):%{time_starttransfer}-%{time_pretransfer}
 3.內容傳輸時間(Content Transfer):%{time_total}-%{time_starttransfer}


如果指令太長,可以把先將curl輸出內容存成檔案(curl-format.txt)
\n
Time_namelookup:  %{time_namelookup}\n
Time_connect:  %{time_connect}\n
Time_appconnect:  %{time_appconnect}\n
Time_redirect:  %{time_redirect}\n
Time_pretransfer:  %{time_pretransfer}\n
Time_starttransfer:  %{time_starttransfer}\n
Time_total:  %{time_total}\n
\n

指令
curl -w @./curl-format.txt -sSo /dev/null https://www.google.com.tw

輸出結果
Time_namelookup:  0.004
Time_connect:  0.013
Time_appconnect:  0.127
Time_redirect:  0.000
Time_pretransfer:  0.127
Time_starttransfer:  0.179
----------
Time_total:  0.182


curl 各項時間參數
time_appconnect
The time, in seconds, it took from the start until the SSL/SSH/etc connect/handshake to the remote host was completed. (Added in 7.19.0)

time_connect
The time, in seconds, it took from the start until the TCP connect to the remote host (or proxy) was completed.

time_namelookup
The time, in seconds, it took from the start until the name resolving was completed.

time_pretransfer
The time, in seconds, it took from the start until the file transfer was just about to begin. This includes all pre-transfer commands and negotiations that are specific to the particular protocol(s) involved.

time_redirect
The time, in seconds, it took for all redirection steps including name lookup, connect, pretransfer and transfer before the final transaction was started. time_redirect shows the complete execution time for multiple redirections. (Added in 7.12.3)

time_starttransfer
The time, in seconds, it took from the start until the first byte was just about to be transferred. This includes time_pretransfer and also the time the server needed to calculate the result.

time_total
The total time, in seconds, that the full operation lasted.

url_effective
The URL that was fetched last. This is most meaningful if you've told curl to follow location: headers.


參考資料:
curl.1 the man page
使用 curl 命令分析请求的耗时情况
CURL 於 CLI 顯示 TCP、SSL 的連線時間
Performance Testing with cURL, Part 1: Basics
Curl 查看網頁載入前的各項時間 (windows)