Notes: 2013

clock_gettime(), clock_getres()

Database Normalization

Database Design - Normalization
http://en.wikipedia.org/wiki/Relational_database_design#Normalization

Database Normalization
http://en.wikipedia.org/wiki/Database_normalization

Denormailization
http://en.wikipedia.org/wiki/Denormalization

internationalization (i18n)

Locale = Language code (ISO-639) + Country code (ISO-3166)
e.g. zh_TW

pip, virtualenv, virtualenvwrapper

pip

安裝easy_install
安裝pip

easy_install pip
利用pip安裝套件, 移除套件

pip [套件名稱]
pip [套件名稱==套件版本]
pip uninstall [套件名稱]

利用pip列出已經安裝的套件

pip freeze

利用pip升級套件

pip install -U [套件名稱]

利用pip搜尋可安裝的套件

pip search [關鍵字]

pip使用方法

pip help

pip封裝相關套件,之後方便使用

e.g. package name => vimpyre
pip bundle vimpyre.pybundle vimpyre #封裝在vimpyre.pybundle
pip install vimpyre.pybundle #令一個環境安裝

pip + virtualenv

安裝virtualenv

pip install virtualenv

建立一個虛擬環境

virtualenv ENV_NAME

初始化虛擬環境(每次都要作一次)

source ./ENV_NAME/bin/activate

在虛擬環境中安裝套件

pip install vimpyre
接著可以在 ./my_python_env/lib/python(版號)/site-packages 發現剛剛安裝的 vimpyre 及其相關套件

退出虛擬環境

deactivate

避免 pip 在沒有進入虛擬環境時被使用

在 ~/.bashrc 加上 export PIP_REQUIRE_VIRTUALENV=true

virtualenvwrapper

安裝virtualenvwrapper

pip install virtualenvwrapper

建立虛擬環境

mkvirtualenv ENV_NAME

複製虛擬環境

cpvirtualenv SRC_NAME TARGET_NAME

移除虛擬環境

rmvirtualenv ENV_NAME

啟動虛擬環境

workon ENV_NAME

列出虛擬環境

lsvirtualenv -b #簡短
lsvirtualenv -l #詳細(預設)
lsvirtualenv -h #help information

離開虛擬環境

deactivate

Postgresql

sudo apt-get install libpq-dev python-dev # Ubuntu

pip install psycopg2==2.5 # Or other edition

Intall pip, virtualenv, virtualenvwrapper on Mac

sudo easy_install pip
sudo pip install virtualenv
sudo pip install virtualenvwrapper

Storage Overview

SAMBA (SMB - Sever Message Block)

一個讓 Windows 系統存取 Linux 檔案系統的 protocol, 鳥哥對此有很好的介紹. 他也提到, CIFS (Common Internet File System) 和 NFS (Network File System) 只在 Windows 或 Unix 裡面互通。不過，NFS 僅能讓 Unix 機器溝通， CIFS 只能讓 Windows 機器溝通。

Scale out & Scale up

Scale Out (Scale horizontally)橫向擴展，向外擴展

e.g. 我們向原有的web, 郵件系統添加一個新機器

Scale Up (Scale vertically)縱向擴展，向上擴展

e.g. 我們向原有的機器添加CPU, 內存

無論是Scale out, Scale up, Scale in，實際上就是一種架構的概念，這些概念可以用在儲存上, 數據庫上, 網路上等。

Storage Virtualization

http://en.wikipedia.org/wiki/Storage_virtualization

Storage Performance Council (SPC-1)

http://www.storageperformance.org/results/benchmark_results_spc1/

Storage Service

Block Level

DAS, SAN

File Level

Block level storage v.s. File level storage

http://www.techrepublic.com/blog/datacenter/block-level-storage-vs-file-level-storage-a-comparison/3766

Storage Architecture

A external storage system directly attached to a server without a storage network in between
Interface : ~~SCSI~~, SAS, , ~~iSCSI~~, FC, ~~PCI-E~~
Block-Level access
~~Internal DAS~~
External DAS

A network of storage devices that can be accessed by multiple servers
Block-Level access
FC SAN
iSCSI SAN
Hybrid SAN
Others

FC-SAN v.s. IP SAN

Performance

High v.s. Low

TCO (Total cost of Ownership)

High v.s. Low

Manager Expertise

High v.s. Low

Host Components

FC HBA v.s. iSCSI HBA or NIC + software initiators

Networking Components

FC Switch, FC Cables v.s. Ethernet Switch, Ethernet Cables

Storage Components

FC-Host Storage v.s. iSCSI Host Storage

RAID Levels

See 儲存產品暨產業基本概要
JBOD (Just a Bunch Of Disks)

http://zh.wikipedia.org/wiki/RAID#JBOD

NRAID (Non-RAID, SPAN)

Link all of the drives in a sequence

RAID 0

Striping

RAID 1

Mirroring

RAID 3

Disk Striping with Dedicated Parity Disk

RAID 5

Disk Striping with Interspersed Parity Disk

RAID 6

Disk Striping with Two Interspersed Parity Disk

RAID 10
RAID 30
RAID 50

http://forum.slime.com.tw/thread209763.html

http://www.prepressure.com/library/technology/raid

Physical Drive, Logical Drive, Logical Volume (Pool)

e.g. A physical drive partitioned into three logical drive C:, D:, E:

http://www.ehow.com/facts_6822924_physical-vs_-logical-hard-drives.html

Logical Drive (LD): Simply an array of independent physical drives

Logical Volume (Pool)(LV) : A LV is composed of one or several logical drives, the member logical drives can be the same RAID level or different RAID levels.

Partitions (Virtual Volume)(VV): The LV can be divided into partitions.

During operation, the hose sees
1) a non-partitioned LV or
2) a partition of a partitioned LV as one single physical drive

Cylinder, Head, Sector

http://en.wikipedia.org/wiki/Cylinder-head-sector

Write Back Cache v.s. Write Through Cache (Write v.s. Read)

WriteThrough:每次遇到寫入指令時都會真正將資料寫入記憶體中
WriteBack:遇到寫入指令時先不將資料寫入記憶體中,而是將 Cache 內儲存的該筆資料標示為已更改,等系統閒置時或該記憶體位址將被其他重複率更高的位址取代時,才真正將資料回寫
WriteThrough 的特點是效率較差,但安全性高,WriteBack 則正好相反反

Master Boot Record (MBR) v.s. GUID Partition Table (GPT)

http://www.petri.co.il/gpt-vs-mbr-based-disks.htm#

Multipath I/O

http://en.wikipedia.org/wiki/Multipath_I/O

Tagged Command Queuing (TCQ)

http://en.wikipedia.org/wiki/Tagged_Command_Queuing

Self-Monitoring, Analysis and Reporting Technology (S.M.A.R.T.)

http://en.wikipedia.org/wiki/S.M.A.R.T.

Fibre Channel

** Ports
=> A port in fibre channel terminology is any entity that actively communicates over the network, not necessarily a hardware port
=> http://en.wikipedia.org/wiki/Fibre_Channel#Ports

Topologies
=> FC_P2P : point to point
=> FC_AL : arbitrated loop
=> FC_SW : swiched fabric

AL_PA: Arbitrated Loop. Physical Address
=> Each device communicating on an arbitrated loop

World Wide Name (WWN)

=> Each device node has a WWNN
=> All ports share the WWNN, and own a unique WWPN
=> World Wide Node Name (WWNN)

=> World Wide Port Name (WWPN)

http://en.wikipedia.org/wiki/World_Wide_Name
http://ithelp.ithome.com.tw/question/10009960

Internet Storage Name Service (iSNS)

http://en.wikipedia.org/wiki/Internet_Storage_Name_Service

Clock synchronization

http://en.wikipedia.org/wiki/Clock_synchronization

SCSI command

http://en.wikipedia.org/wiki/SCSI_command

Bus, SAN, DAS, and NAS

SCSI (Small Computer System Interface)(並列SCSI)

SCSI initiator and target

Initiator : Host (computer), client
Target : Data storage device (HDD), server
Initiator and Target : mainly for communicating with each other
Target includes LUN (Logic Units) : mainly for execution command
Reference: http://en.wikipedia.org/wiki/SCSI_initiator_and_target

trminator :
Reference: http://en.wikipedia.org/wiki/SCSI_terminator

High voltage v.s. Low voltage

The difference between low voltage and high voltage for the same size cable is that cable size affects current carrying capacity, while power is voltage times current. As a result, the same size cable can transfer more power at high voltage than at low voltage. (Power = Voltage * Current, P = V * I)

This is why utility power transmission and distribution systems use high voltage. A typical transmission voltage in the US might be 69 kv, 138 kv, even 345 kv, and a typical distribution voltage in the US might by 13.2 kv.

Synchronous & Asynchronous Data transfer

Reference: http://www.ehow.com/facts_7463933_synchronous-asynchronous-data-transfer.html

SAS (Serial Attached SCSI)(串列SCSI)

由並列SCSI物理存儲介面演化而來。與並列方式相比，串列方式提供更快速的通訊傳輸速度以及更簡易的配置。此外SAS支援與串列ATA設備相容，且兩者可以使用相類似的電纜。SATA的硬碟可接在SAS的控制器使用，但SAS硬碟並不能接在SATA的控制器使用。

SAS是點對點（point-to-point）連接，並允許多個埠集中於單個控制器上，可以建立在主機板也可以另外添加。該技術建立在強大的並列SCSI通訊技術基礎上。SAS是採用SATA相容的電纜線採取點對點連接方式，從而在計算機系統中不需要建立菊花鏈（daisy-chaining）方式便可簡單地實現線纜安裝。

第一代SAS為陣列中的每個驅動器提供3.0 Gbps（約300 MBps）的傳輸速率。
第二代SAS為陣列中的每個驅動器提供6.0 Gbps（約600 MBps）的傳輸速率。

SAS由3種型別協定組成，根據連線的不同裝置使用相應的協定進行資料傳輸。

序列SCSI協定 (SSP) — 用於傳輸SCSI命令。
SATA通道協定 (STP) — 用於傳輸SATA資料。
SCSI管理協定 (SMP) — 用於對SAS裝置的維護和管理。

iSCSI (發音為 /аɪskʌzi/)(IP-SAN = IP-Storage Area Network)

與傳統的SCSI技術比較起來，iSCSI技術有以下三個革命性的變化：

把原來只用於本機的SCSI協同透過TCP/IP網路傳送，使連接距離可作無限的地域延伸。
連接的伺服器數量無限（原來的SCSI-3的上限是15）。
由於是伺服器架構，因此也可以實現在線擴容以至動態部署。

MPIO & MCS
http://blog.xuite.net/weirchen/blog/22184665-%E6%B7%B1%E5%85%A5%E4%BA%86%E8%A7%A3+iSCSI%E7%9A%84%E5%85%A9%E7%A8%AE%E5%A4%9A%E9%87%8D%E8%B7%AF%E5%BE%91%E5%AD%98%E5%8F%96%E6%A9%9F%E5%88%B6-2009.05.18%E5%A2%9E%E4%BF%AE

ATA (Advanced Technology Attachment)

(PATA = Parallel ATA)或稱(IDE = Integrated Drive Electronics)

SATA (Serial ATA)

是串列SCSI (SAS) 的孿生兄弟，兩者的排線相容，SATA硬碟可接上SAS介面。它是一種電腦匯流排，主要功能是用作主機板和大量儲存裝置（如硬碟及光碟機）之間的資料傳輸之用。SATA是已經完全取代舊式PATA（Parallel ATA或稱IDE）的新型硬碟介面，因採用序列方式傳輸資料而得名。現時，SATA分別有SATA 1.5Gbit/s、SATA 3Gbit/s和SATA 6Gbit/s三種規格。

主機板上的SATA 1.5Gb/s插口

頻寬速度換算

需要注意的是，在一些新的技術標準中，為了防止資料在高速傳輸中出錯而加入了校驗碼，比如PCI-E 2.0、USB 3.0和SATA 3.0中採用的是8/10編碼，每10位編碼中只有8位元是真實資料，這時單位換算就不再是1:8而是1:10了，USB 3.0的5Gbps速度實際上是理論500MB/s而非理論625MB/s，SATA 6Gbps的理論速度則是600MB/s而非750MB/s。所謂3Gb/s的演算法，3000MHz的頻率 x 每次發送一個資料 x 80%(8b/10b的編碼) / 8 bits per byte = 300Mbytes/s，同理1.5Gb/s也是這樣可算成150MB/s，也就是一般我們在買硬碟時，有時候會看到SATA 150MB/s / 300MB/s，有時候又會看到SATA 1.5Gb/s / 3Gb/s的緣故。

以USB 3.0而言，它擁有5Gbps的頻寬，每次發送一個資料 x 80%(8b/10b的編碼) / 8 bits per byte = 500Mbytes/s，所以USB 3.0的頻寬比SATA 3.0的600MB/s 還來的小。

SATA版本	頻寬	速度
SATA 3.0	6Gb/s	600MB/s
SATA 2.0	3Gb/s	300MB/s
SATA 1.0	1.5Gb/s	150MB/s

USB版本	頻寬	速度
USB 3.0	5Gb/s	500MB/s

Fibre Channel

http://en.wikipedia.org/wiki/Fibre_Channel

DAS

Refers to a digital storage system directly attached to a server or workstation, without a storage network in between.It is a retronym, mainly used to differentiate non-networked storage from SAN and NAS.

A typical DAS system is made of a data storage device (for example enclosures holding a number of hard disk drives) connected directly to a computer through a host bus adapter (HBA). Between those two points there is no network device (like hub, switch, or router), and this is the main characteristic of DAS.

The main protocols used for DAS connections are ATA, SATA, eSATA,^[1] SCSI, SAS, and Fibre Channel.

SAN

出於歷史原因，數據中心中最初都是SCSI 磁碟陣列的「孤島」群。每個單獨的小「島嶼」都是一個專門的直接連接存儲器應用，並且被視作無數個「虛擬硬碟驅動器」（例如LUNs）。本質上來說，SAN就是將一個個存儲「孤島」使用高速網路連接起來，這樣使得所有的應用可以訪問所有的磁碟。

作業系統會將SAN視為一組LUN，並且在LUN上維護自己的File System。這些不能在多個作業系統/主機之間進行共享的本地文件系統，具有非常高的可靠性和十分廣泛的應用。如果兩個獨立的本地文件系統存在於一個共享的LUN上，它們彼此沒有任何機制來知道對方的存在，沒有類似緩存同步(Distributed Lock Manager)的機制，所以可能發生數據丟失的情況。因此，在主機之間通過SAN共享數據，需要一些複雜的高級解決方案，例如SAN文件系統或者計算機集群。撇開這些問題，SAN對於提高存儲能力的應用有很大幫助，因為多個伺服器可以共享磁碟陣列上的存儲空間。SAN的一項典型應用是需要高速塊級別訪問的數據操作伺服器，比如電子郵件伺服器、資料庫、高利用率的文件伺服器等。

NAS

相對地，NAS允許多台計算機經過網路訪問同一個File system(Storage)，並且會自動同步它們的操作。由於NAS的引入使得SAN存儲可以被容易地轉換為NAS。

DAS、NAS 和 SAN 的比較

Daisy-chaining (菊花鏈)

Except for star-based networks, the easiest way to add more computers into a network is by daisy-chaining, or connecting each computer in series to the next. If a message is intended for a computer partway down the line, each system bounces it along in sequence until it reaches the destination. A daisy-chained network can take two basic forms: linear and ring.

In electrical and electronic engineering a daisy chain is a wiring scheme in which multiple devices are wired together in sequence or in a ring. Other than a full, single loop, systems which contain internal loops cannot be called daisy chains.

Linear Topology
Ring Topology

名稱	頻寬 (Mbit/s)	速度 (MB/s)	接線最大長度 (m)	電源供應	每頻道最多可接設備
eSATA	3,000	300	2 with eSATA HBA (1 with passive adapter)	否 ^[2]	1（15 with port multiplier）
串列SCSI	3,000	375	8	否	4
SATA 6.0Gb/s	6,000	600	1
SATA 3.0Gb/s	3,000	300	1	否	1（15 with port multiplier）
SATA 1.5Gb/s	1,500	150	1	否	1 per line
PATA 133	1,064	133	0.46（18 英寸）	否	2
FireWire 3200	3,144	393	100; alternate cables available for 100 m+	15 W, 12–25 V	63 (with hub)
FireWire 800	786	98.25	100^[3]	15 W, 12–25 V	63 (with hub)
FireWire 400	393	49.13	4.5^[3]^[4]	15 W, 12–25 V	63 (with hub)
USB 3.0	5,000	500		4.5 W, 5 V
USB 2.0	480	60	5^[5]	2.5 W, 5 V	127 (with hub)
USB 1.1	12	1.5
USB 1.0	1.5	0.1875
Ultra-640 SCSI	5,120	640	12	否	15 (plus the HBA)
Ultra-320 SCSI	2,560	320
Fiber Channel 透過銅線	4,000	400	12	否	126 （16777216 with switches）
Fiber Channel 透過光纖	10,520	2000	2–50000	否	126 （16777216 with switches）
Infiniband 12X Quad-rate	120,000	12,000	5 （銅線）^[6]^[7] <10000 nbsp="" p="">	否	1 with Point to point Many with switched fabric

The OpenMP Manual

The GNU OpenMP Manual

http://gcc.gnu.org/onlinedocs/libgomp/

Parallelization of Codes on the SGI Origins

Reference:

http://people.nas.nasa.gov/~schang/origin_parallel.html

grep, awk, sed, regular expression

== grep ==

# n=列出行數
grep -nr ''

# 列出找到的以下30行
grep -A 30 ....

== awk ==

awk「樣式」檔案：把符合樣式的資料列顯示出來。
awk '{動作}' 檔案：對每一列都執行{}中的動作。
awk '樣式{動作}' 檔案：對符合樣式的資料列，執行{}中的動作。

awk 的用法1：
awk '/La/' dataf3
顯示含La 的資料列。

awk 的用法2：
awk '{ print $1, $2 }' dataf3
顯示dataf3 每一列的第1 和第2 個欄位。
$1 代表第1 個欄位，$2 代表第二欄位，其他依此類推。

awk 的用法3：
awk '/La/{ print $1, $2 }' dataf3
將含有La 關鍵字的資料列的第1 及第2 個欄位顯示出來。

awk 的用法4：
awk -F: '/^ols3/{ print $3, $4 }' /etc/passwd
使用選項 -F，指定:為分隔字元，帳號ols3 的uid（第3 欄位）及gid（第4
欄位）顯示出來。

awk 的用法5：
awk -F: 'BEGIN{OFS="+++"}/^ols3/{ print $1, $2, $3, $4, $5 }' /etc/passwd
以:為分隔字元，+++為輸出欄位分隔符號，將帳號ols3 的第1~5 欄顯示出來。

awk 的用法6：
cat filename | awk '{ sum+=$3} END {print THE SUM = sum}'

== sed == stream editor
Reference:http://sed.sourceforge.net/sed1line.txt

USEFUL ONE-LINE SCRIPTS FOR SED (Unix stream editor)        Dec. 29, 2005
Compiled by Eric Pement - pemente[at]northpark[dot]edu        version 5.5

Latest version of this file (in English) is usually at:
   http://sed.sourceforge.net/sed1line.txt
   http://www.pement.org/sed/sed1line.txt

This file will also available in other languages:
  Chinese     - http://sed.sourceforge.net/sed1line_zh-CN.html
  Czech       - http://sed.sourceforge.net/sed1line_cz.html
  Dutch       - http://sed.sourceforge.net/sed1line_nl.html
  French      - http://sed.sourceforge.net/sed1line_fr.html
  German      - http://sed.sourceforge.net/sed1line_de.html
  Italian     - (pending)
  Portuguese  - http://sed.sourceforge.net/sed1line_pt-BR.html
  Spanish     - (pending)


FILE SPACING:

 # double space a file
 sed G

 # double space a file which already has blank lines in it. Output file
 # should contain no more than one blank line between lines of text.
 sed '/^$/d;G'

 # triple space a file
 sed 'G;G'

 # undo double-spacing (assumes even-numbered lines are always blank)
 sed 'n;d'

 # insert a blank line above every line which matches "regex"
 sed '/regex/{x;p;x;}'

 # insert a blank line below every line which matches "regex"
 sed '/regex/G'

 # insert a blank line above and below every line which matches "regex"
 sed '/regex/{x;p;x;G;}'

NUMBERING:

 # number each line of a file (simple left alignment). Using a tab (see
 # note on '\t' at end of file) instead of space will preserve margins.
 sed = filename | sed 'N;s/\n/\t/'

 # number each line of a file (number on left, right-aligned)
 sed = filename | sed 'N; s/^/     /; s/ *\(.\{6,\}\)\n/\1  /'

 # number each line of file, but only print numbers if line is not blank
 sed '/./=' filename | sed '/./N; s/\n/ /'

 # count lines (emulates "wc -l")
 sed -n '$='

TEXT CONVERSION AND SUBSTITUTION:

 # IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format.
 sed 's/.$//'               # assumes that all lines end with CR/LF
 sed 's/^M$//'              # in bash/tcsh, press Ctrl-V then Ctrl-M
 sed 's/\x0D$//'            # works on ssed, gsed 3.02.80 or higher

 # IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format.
 sed "s/$/`echo -e \\\r`/"            # command line under ksh
 sed 's/$'"/`echo \\\r`/"             # command line under bash
 sed "s/$/`echo \\\r`/"               # command line under zsh
 sed 's/$/\r/'                        # gsed 3.02.80 or higher

 # IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format.
 sed "s/$//"                          # method 1
 sed -n p                             # method 2

 # IN DOS ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format.
 # Can only be done with UnxUtils sed, version 4.0.7 or higher. The
 # UnxUtils version can be identified by the custom "--text" switch
 # which appears when you use the "--help" switch. Otherwise, changing
 # DOS newlines to Unix newlines cannot be done with sed in a DOS
 # environment. Use "tr" instead.
 sed "s/\r//" infile >outfile         # UnxUtils sed v4.0.7 or higher
 tr -d \r outfile            # GNU tr version 1.22 or higher

 # delete leading whitespace (spaces, tabs) from front of each line
 # aligns all text flush left
 sed 's/^[ \t]*//'                    # see note on '\t' at end of file

 # delete trailing whitespace (spaces, tabs) from end of each line
 sed 's/[ \t]*$//'                    # see note on '\t' at end of file

 # delete BOTH leading and trailing whitespace from each line
 sed 's/^[ \t]*//;s/[ \t]*$//'

 # insert 5 blank spaces at beginning of each line (make page offset)
 sed 's/^/     /'

 # align all text flush right on a 79-column width
 sed -e :a -e 's/^.\{1,78\}$/ &/;ta'  # set at 78 plus 1 space

 # center all text in the middle of 79-column width. In method 1,
 # spaces at the beginning of the line are significant, and trailing
 # spaces are appended at the end of the line. In method 2, spaces at
 # the beginning of the line are discarded in centering the line, and
 # no trailing spaces appear at the end of lines.
 sed  -e :a -e 's/^.\{1,77\}$/ & /;ta'                     # method 1
 sed  -e :a -e 's/^.\{1,77\}$/ &/;ta' -e 's/\( *\)\1/\1/'  # method 2

 # substitute (find and replace) "foo" with "bar" on each line
 sed 's/foo/bar/'             # replaces only 1st instance in a line
 sed 's/foo/bar/4'            # replaces only 4th instance in a line
 sed 's/foo/bar/g'            # replaces ALL instances in a line
 sed 's/\(.*\)foo\(.*foo\)/\1bar\2/' # replace the next-to-last case
 sed 's/\(.*\)foo/\1bar/'            # replace only the last case

 # substitute "foo" with "bar" ONLY for lines which contain "baz"
 sed '/baz/s/foo/bar/g'

 # substitute "foo" with "bar" EXCEPT for lines which contain "baz"
 sed '/baz/!s/foo/bar/g'

 # change "scarlet" or "ruby" or "puce" to "red"
 sed 's/scarlet/red/g;s/ruby/red/g;s/puce/red/g'   # most seds
 gsed 's/scarlet\|ruby\|puce/red/g'                # GNU sed only

 # reverse order of lines (emulates "tac")
 # bug/feature in HHsed v1.5 causes blank lines to be deleted
 sed '1!G;h;$!d'               # method 1
 sed -n '1!G;h;$p'             # method 2

 # reverse each character on the line (emulates "rev")
 sed '/\n/!G;s/\(.\)\(.*\n\)/&\2\1/;//D;s/.//'

 # join pairs of lines side-by-side (like "paste")
 sed '$!N;s/\n/ /'

 # if a line ends with a backslash, append the next line to it
 sed -e :a -e '/\\$/N; s/\\\n//; ta'

 # if a line begins with an equal sign, append it to the previous line
 # and replace the "=" with a single space
 sed -e :a -e '$!N;s/\n=/ /;ta' -e 'P;D'

 # add commas to numeric strings, changing "1234567" to "1,234,567"
 gsed ':a;s/\B[0-9]\{3\}\>/,&/;ta'                     # GNU sed
 sed -e :a -e 's/\(.*[0-9]\)\([0-9]\{3\}\)/\1,\2/;ta'  # other seds

 # add commas to numbers with decimal points and minus signs (GNU sed)
 gsed -r ':a;s/(^|[^0-9.])([0-9]+)([0-9]{3})/\1\2,\3/g;ta'

 # add a blank line every 5 lines (after lines 5, 10, 15, 20, etc.)
 gsed '0~5G'                  # GNU sed only
 sed 'n;n;n;n;G;'             # other seds

SELECTIVE PRINTING OF CERTAIN LINES:

 # print first 10 lines of file (emulates behavior of "head")
 sed 10q

 # print first line of file (emulates "head -1")
 sed q

 # print the last 10 lines of a file (emulates "tail")
 sed -e :a -e '$q;N;11,$D;ba'

 # print the last 2 lines of a file (emulates "tail -2")
 sed '$!N;$!D'

 # print the last line of a file (emulates "tail -1")
 sed '$!d'                    # method 1
 sed -n '$p'                  # method 2

 # print the next-to-the-last line of a file
 sed -e '$!{h;d;}' -e x              # for 1-line files, print blank line
 sed -e '1{$q;}' -e '$!{h;d;}' -e x  # for 1-line files, print the line
 sed -e '1{$d;}' -e '$!{h;d;}' -e x  # for 1-line files, print nothing

 # print only lines which match regular expression (emulates "grep")
 sed -n '/regexp/p'           # method 1
 sed '/regexp/!d'             # method 2

 # print only lines which do NOT match regexp (emulates "grep -v")
 sed -n '/regexp/!p'          # method 1, corresponds to above
 sed '/regexp/d'              # method 2, simpler syntax

 # print the line immediately before a regexp, but not the line
 # containing the regexp
 sed -n '/regexp/{g;1!p;};h'

 # print the line immediately after a regexp, but not the line
 # containing the regexp
 sed -n '/regexp/{n;p;}'

 # print 1 line of context before and after regexp, with line number
 # indicating where the regexp occurred (similar to "grep -A1 -B1")
 sed -n -e '/regexp/{=;x;1!p;g;$!N;p;D;}' -e h

 # grep for AAA and BBB and CCC (in any order)
 sed '/AAA/!d; /BBB/!d; /CCC/!d'

 # grep for AAA and BBB and CCC (in that order)
 sed '/AAA.*BBB.*CCC/!d'

 # grep for AAA or BBB or CCC (emulates "egrep")
 sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d    # most seds
 gsed '/AAA\|BBB\|CCC/!d'                        # GNU sed only

 # print paragraph if it contains AAA (blank lines separate paragraphs)
 # HHsed v1.5 must insert a 'G;' after 'x;' in the next 3 scripts below
 sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;'

 # print paragraph if it contains AAA and BBB and CCC (in any order)
 sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;/BBB/!d;/CCC/!d'

 # print paragraph if it contains AAA or BBB or CCC
 sed -e '/./{H;$!d;}' -e 'x;/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d
 gsed '/./{H;$!d;};x;/AAA\|BBB\|CCC/b;d'         # GNU sed only

 # print only lines of 65 characters or longer
 sed -n '/^.\{65\}/p'

 # print only lines of less than 65 characters
 sed -n '/^.\{65\}/!p'        # method 1, corresponds to above
 sed '/^.\{65\}/d'            # method 2, simpler syntax

 # print section of file from regular expression to end of file
 sed -n '/regexp/,$p'

 # print section of file based on line numbers (lines 8-12, inclusive)
 sed -n '8,12p'               # method 1
 sed '8,12!d'                 # method 2

 # print line number 52
 sed -n '52p'                 # method 1
 sed '52!d'                   # method 2
 sed '52q;d'                  # method 3, efficient on large files

 # beginning at line 3, print every 7th line
 gsed -n '3~7p'               # GNU sed only
 sed -n '3,${p;n;n;n;n;n;n;}' # other seds

 # print section of file between two regular expressions (inclusive)
 sed -n '/Iowa/,/Montana/p'             # case sensitive

SELECTIVE DELETION OF CERTAIN LINES:

 # print all of file EXCEPT section between 2 regular expressions
 sed '/Iowa/,/Montana/d'

 # delete duplicate, consecutive lines from a file (emulates "uniq").
 # First line in a set of duplicate lines is kept, rest are deleted.
 sed '$!N; /^\(.*\)\n\1$/!P; D'

 # delete duplicate, nonconsecutive lines from a file. Beware not to
 # overflow the buffer size of the hold space, or else use GNU sed.
 sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'

 # delete all lines except duplicate lines (emulates "uniq -d").
 sed '$!N; s/^\(.*\)\n\1$/\1/; t; D'

 # delete the first 10 lines of a file
 sed '1,10d'

 # delete the last line of a file
 sed '$d'

 # delete the last 2 lines of a file
 sed 'N;$!P;$!D;$d'

 # delete the last 10 lines of a file
 sed -e :a -e '$d;N;2,10ba' -e 'P;D'   # method 1
 sed -n -e :a -e '1,10!{P;N;D;};N;ba'  # method 2

 # delete every 8th line
 gsed '0~8d'                           # GNU sed only
 sed 'n;n;n;n;n;n;n;d;'                # other seds

 # delete lines matching pattern
 sed '/pattern/d'

 # delete ALL blank lines from a file (same as "grep '.' ")
 sed '/^$/d'                           # method 1
 sed '/./!d'                           # method 2

 # delete all CONSECUTIVE blank lines from file except the first; also
 # deletes all blank lines from top and end of file (emulates "cat -s")
 sed '/./,/^$/!d'          # method 1, allows 0 blanks at top, 1 at EOF
 sed '/^$/N;/\n$/D'        # method 2, allows 1 blank at top, 0 at EOF

 # delete all CONSECUTIVE blank lines from file except the first 2:
 sed '/^$/N;/\n$/N;//D'

 # delete all leading blank lines at top of file
 sed '/./,$!d'

 # delete all trailing blank lines at end of file
 sed -e :a -e '/^\n*$/{$d;N;ba' -e '}'  # works on all seds
 sed -e :a -e '/^\n*$/N;/\n$/ba'        # ditto, except for gsed 3.02.*

 # delete the last line of each paragraph
 sed -n '/^$/{p;h;};/./{x;/./p;}'

SPECIAL APPLICATIONS:

 # remove nroff overstrikes (char, backspace) from man pages. The 'echo'
 # command may need an -e switch if you use Unix System V or bash shell.
 sed "s/.`echo \\\b`//g"    # double quotes required for Unix environment
 sed 's/.^H//g'             # in bash/tcsh, press Ctrl-V and then Ctrl-H
 sed 's/.\x08//g'           # hex expression for sed 1.5, GNU sed, ssed

 # get Usenet/e-mail message header
 sed '/^$/q'                # deletes everything after first blank line

 # get Usenet/e-mail message body
 sed '1,/^$/d'              # deletes everything up to first blank line

 # get Subject header, but remove initial "Subject: " portion
 sed '/^Subject: */!d; s///;q'

 # get return address header
 sed '/^Reply-To:/q; /^From:/h; /./d;g;q'

 # parse out the address proper. Pulls out the e-mail address by itself
 # from the 1-line return address header (see preceding script)
 sed 's/ *(.*)//; s/>.*//; s/.*[:<] *//'

 # add a leading angle bracket and space to each line (quote a message)
 sed 's/^/> /'

 # delete leading angle bracket & space from each line (unquote a message)
 sed 's/^> //'

 # remove most HTML tags (accommodates multiple-line tags)
 sed -e :a -e 's/<[^>]*>//g;/

zipup.bat dir /b *.txt | sed "s/^$.*$\.TXT/pkzip -mo \1 \1.TXT/" >>zipup.bat TYPICAL USE: Sed takes one or more editing commands and applies all of them, in sequence, to each line of input. After all the commands have been applied to the first input line, that line is output and a second input line is taken for processing, and the cycle repeats. The preceding examples assume that input comes from the standard input device (i.e, the console, normally this will be piped input). One or more filenames can be appended to the command line if the input does not come from stdin. Output is sent to stdout (the screen). Thus: cat filename | sed '10q' # uses piped input sed '10q' filename # same effect, avoids a useless "cat" sed '10q' filename > newfile # redirects output to disk For additional syntax instructions, including the way to apply editing commands from a disk file instead of the command line, consult "sed & awk, 2nd Edition," by Dale Dougherty and Arnold Robbins (O'Reilly, 1997; http://www.ora.com), "UNIX Text Processing," by Dale Dougherty and Tim O'Reilly (Hayden Books, 1987) or the tutorials by Mike Arst distributed in U-SEDIT2.ZIP (many sites). To fully exploit the power of sed, one must understand "regular expressions." For this, see "Mastering Regular Expressions" by Jeffrey Friedl (O'Reilly, 1997). The manual ("man") pages on Unix systems may be helpful (try "man sed", "man regexp", or the subsection on regular expressions in "man ed"), but man pages are notoriously difficult. They are not written to teach sed use or regexps to first-time users, but as a reference text for those already acquainted with these tools. QUOTING SYNTAX: The preceding examples use single quotes ('...') instead of double quotes ("...") to enclose editing commands, since sed is typically used on a Unix platform. Single quotes prevent the Unix shell from intrepreting the dollar sign ($) and backquotes (`...`), which are expanded by the shell if they are enclosed in double quotes. Users of the "csh" shell and derivatives will also need to quote the exclamation mark (!) with the backslash (i.e., \!) to properly run the examples listed above, even within single quotes. Versions of sed written for DOS invariably require double quotes ("...") instead of single quotes to enclose editing commands. USE OF '\t' IN SED SCRIPTS: For clarity in documentation, we have used the expression '\t' to indicate a tab character (0x09) in the scripts. However, most versions of sed do not recognize the '\t' abbreviation, so when typing these scripts from the command line, you should press the TAB key instead. '\t' is supported as a regular expression metacharacter in awk, perl, and HHsed, sedmod, and GNU sed v3.02.80. VERSIONS OF SED: Versions of sed do differ, and some slight syntax variation is to be expected. In particular, most do not support the use of labels (:name) or branch instructions (b,t) within editing commands, except at the end of those commands. We have used the syntax which will be portable to most users of sed, even though the popular GNU versions of sed allow a more succinct syntax. When the reader sees a fairly long command such as this: sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d it is heartening to know that GNU sed will let you reduce it to: sed '/AAA/b;/BBB/b;/CCC/b;d' # or even sed '/AAA\|BBB\|CCC/b;d' In addition, remember that while many versions of sed accept a command like "/one/ s/RE1/RE2/", some do NOT allow "/one/! s/RE1/RE2/", which contains space before the 's'. Omit the space when typing the command. OPTIMIZING FOR SPEED: If execution speed needs to be increased (due to large input files or slow processors or hard disks), substitution will be executed more quickly if the "find" expression is specified before giving the "s/.../.../" instruction. Thus: sed 's/foo/bar/g' filename # standard replace command sed '/foo/ s/foo/bar/g' filename # executes more quickly sed '/foo/ s//bar/g' filename # shorthand sed syntax On line selection or deletion in which you only need to output lines from the first part of the file, a "quit" command (q) in the script will drastically reduce processing time for large files. Thus: sed -n '45,50p' filename # print line nos. 45-50 of a file sed -n '51q;45,50p' filename # same, but executes much faster If you have any additional scripts to contribute or if you find errors in this document, please send e-mail to the compiler. Indicate the version of sed you used, the operating system it was compiled for, and the nature of the problem. To qualify as a one-liner, the command line must be 65 characters or less. Various scripts in this file have been written or contributed by: Al Aab # founder of "seders" list Edgar Allen # various Yiorgos Adamopoulos # various Dale Dougherty # author of "sed & awk" Carlos Duarte # author of "do it with sed" Eric Pement # author of this document Ken Pizzini # author of GNU sed v3.02 S.G. Ravenhall # great de-html script Greg Ubben # many contributions & much help -------------------------------------------------------------------------
== regular expression ==

1) . 代表任意的字元。
# data\.... 代表 data. 後接三個字元，如data.txt、data.cfg

2)^ 在列首, $ 在尾部
# ^Jack，代表Jack 應出現在列首; 樣式123$，代表在列的最後面是123。

3) [...] 字元集合
# [Ss]ame 代表Same 或same。
# [A-Z] 一個大寫字母。
# [a-z] 一個小寫字母。
# [0-9] 一個數字。
# [^A-Z] 除了大寫字母之外的一個字元。
# [^a-zA-Z] 一個非英文字母的字元。
# [^a-zA-Z0-9] 一個非英文字母、且非數字的字元。

4) * 代表前面的（左鄰）字元有0 個或0 個以上。+ 代表前面的（左鄰）字元有0 個以上。

5) \{...\} 指定前面的（左鄰）字元的個數。
# [a-z]\{3,5\}代表以小寫字母組成的字串，長度是3~5。

6) $...$ 把比對符合的字串暫時保存起來
# H$...$y 表示要保存H 和y 之間的三個字元。若欲取用保存的字串，可用位置參數，\1 代表第一個保存的字串，\2 代表第二個，其他依此類推。

RewriteEngine in Apache

1)

$ a2enmod rewrite

2)

$ /etc/init.d/apache2 restart

3)


    RewriteEngine On
    RewriteBase /
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteRule . /index.php [L]

Computer architecture simulator

scope: micro-architecture vs. full-system simulators.

The modeled scope could be only one microprocessor or the whole computer system.

Detail: functional vs. timing(performance) simulators
...

input(workload): trace-driven(event-driven) vs execution-driven

trace-driven:

Trace driven simulation is getting the actual data from a real experiment and feeding it to the simulation. The disadvantage is every time the experiment has to be done which is a tedious job, advantage is absolute results.

event-driven:

Memory hierarchy simulator returns simulated time information to reference generator, which is used to schedule simulated processes.

Reference:http://www.cs.berkeley.edu/~culler/cs258-s99/slides/lec08/sld028.htm

Reference:http://en.wikipedia.org/wiki/Computer_architecture_simulator

相關文件

pip

pip + virtualenv

virtualenvwrapper

Postgresql

Intall pip, virtualenv, virtualenvwrapper on Mac

相關文件

Computer Data Storage

Parity bit

Single-ended signaling v.s. Differential signaling

SAMBA (SMB - Sever Message Block)

Scale out & Scale up

Storage Virtualization

Storage Performance Council (SPC-1)

Storage Service

Storage Architecture

RAID Levels

Physical Drive, Logical Drive, Logical Volume (Pool)

Cylinder, Head, Sector

Write Back Cache v.s. Write Through Cache (Write v.s. Read)

Master Boot Record (MBR) v.s. GUID Partition Table (GPT)

Multipath I/O

Tagged Command Queuing (TCQ)

Self-Monitoring, Analysis and Reporting Technology (S.M.A.R.T.)

Fibre Channel

World Wide Name (WWN)

Internet Storage Name Service (iSNS)

Clock synchronization

SCSI command

相關文件

SCSI (Small Computer System Interface)(並列SCSI)

SCSI initiator and target

SAS (Serial Attached SCSI)(串列SCSI)

iSCSI (發音為 /аɪskʌzi/)(IP-SAN = IP-Storage Area Network)

ATA (Advanced Technology Attachment)

SATA (Serial ATA)

頻寬速度換算

Fibre Channel

DAS

SAN

NAS

Daisy-chaining (菊花鏈)

相關文件

The GNU OpenMP Manual