Notes: 2013.10

pip, virtualenv, virtualenvwrapper

pip

安裝easy_install
安裝pip

easy_install pip
利用pip安裝套件, 移除套件

pip [套件名稱]
pip [套件名稱==套件版本]
pip uninstall [套件名稱]

利用pip列出已經安裝的套件

pip freeze

利用pip升級套件

pip install -U [套件名稱]

利用pip搜尋可安裝的套件

pip search [關鍵字]

pip使用方法

pip help

pip封裝相關套件,之後方便使用

e.g. package name => vimpyre
pip bundle vimpyre.pybundle vimpyre #封裝在vimpyre.pybundle
pip install vimpyre.pybundle #令一個環境安裝

pip + virtualenv

安裝virtualenv

pip install virtualenv

建立一個虛擬環境

virtualenv ENV_NAME

初始化虛擬環境(每次都要作一次)

source ./ENV_NAME/bin/activate

在虛擬環境中安裝套件

pip install vimpyre
接著可以在 ./my_python_env/lib/python(版號)/site-packages 發現剛剛安裝的 vimpyre 及其相關套件

退出虛擬環境

deactivate

避免 pip 在沒有進入虛擬環境時被使用

在 ~/.bashrc 加上 export PIP_REQUIRE_VIRTUALENV=true

virtualenvwrapper

安裝virtualenvwrapper

pip install virtualenvwrapper

建立虛擬環境

mkvirtualenv ENV_NAME

複製虛擬環境

cpvirtualenv SRC_NAME TARGET_NAME

移除虛擬環境

rmvirtualenv ENV_NAME

啟動虛擬環境

workon ENV_NAME

列出虛擬環境

lsvirtualenv -b #簡短
lsvirtualenv -l #詳細(預設)
lsvirtualenv -h #help information

離開虛擬環境

deactivate

Postgresql

sudo apt-get install libpq-dev python-dev # Ubuntu

pip install psycopg2==2.5 # Or other edition

Intall pip, virtualenv, virtualenvwrapper on Mac

sudo easy_install pip
sudo pip install virtualenv
sudo pip install virtualenvwrapper

Storage Overview

SAMBA (SMB - Sever Message Block)

一個讓 Windows 系統存取 Linux 檔案系統的 protocol, 鳥哥對此有很好的介紹. 他也提到, CIFS (Common Internet File System) 和 NFS (Network File System) 只在 Windows 或 Unix 裡面互通。不過，NFS 僅能讓 Unix 機器溝通， CIFS 只能讓 Windows 機器溝通。

Scale out & Scale up

Scale Out (Scale horizontally)橫向擴展，向外擴展

e.g. 我們向原有的web, 郵件系統添加一個新機器

Scale Up (Scale vertically)縱向擴展，向上擴展

e.g. 我們向原有的機器添加CPU, 內存

無論是Scale out, Scale up, Scale in，實際上就是一種架構的概念，這些概念可以用在儲存上, 數據庫上, 網路上等。

Storage Virtualization

http://en.wikipedia.org/wiki/Storage_virtualization

Storage Performance Council (SPC-1)

http://www.storageperformance.org/results/benchmark_results_spc1/

Storage Service

Block Level

DAS, SAN

File Level

Block level storage v.s. File level storage

http://www.techrepublic.com/blog/datacenter/block-level-storage-vs-file-level-storage-a-comparison/3766

Storage Architecture

A external storage system directly attached to a server without a storage network in between
Interface : ~~SCSI~~, SAS, , ~~iSCSI~~, FC, ~~PCI-E~~
Block-Level access
~~Internal DAS~~
External DAS

A network of storage devices that can be accessed by multiple servers
Block-Level access
FC SAN
iSCSI SAN
Hybrid SAN
Others

FC-SAN v.s. IP SAN

Performance

High v.s. Low

TCO (Total cost of Ownership)

High v.s. Low

Manager Expertise

High v.s. Low

Host Components

FC HBA v.s. iSCSI HBA or NIC + software initiators

Networking Components

FC Switch, FC Cables v.s. Ethernet Switch, Ethernet Cables

Storage Components

FC-Host Storage v.s. iSCSI Host Storage

RAID Levels

See 儲存產品暨產業基本概要
JBOD (Just a Bunch Of Disks)

http://zh.wikipedia.org/wiki/RAID#JBOD

NRAID (Non-RAID, SPAN)

Link all of the drives in a sequence

RAID 0

Striping

RAID 1

Mirroring

RAID 3

Disk Striping with Dedicated Parity Disk

RAID 5

Disk Striping with Interspersed Parity Disk

RAID 6

Disk Striping with Two Interspersed Parity Disk

RAID 10
RAID 30
RAID 50

http://forum.slime.com.tw/thread209763.html

http://www.prepressure.com/library/technology/raid

Physical Drive, Logical Drive, Logical Volume (Pool)

e.g. A physical drive partitioned into three logical drive C:, D:, E:

http://www.ehow.com/facts_6822924_physical-vs_-logical-hard-drives.html

Logical Drive (LD): Simply an array of independent physical drives

Logical Volume (Pool)(LV) : A LV is composed of one or several logical drives, the member logical drives can be the same RAID level or different RAID levels.

Partitions (Virtual Volume)(VV): The LV can be divided into partitions.

During operation, the hose sees
1) a non-partitioned LV or
2) a partition of a partitioned LV as one single physical drive

Cylinder, Head, Sector

http://en.wikipedia.org/wiki/Cylinder-head-sector

Write Back Cache v.s. Write Through Cache (Write v.s. Read)

WriteThrough:每次遇到寫入指令時都會真正將資料寫入記憶體中
WriteBack:遇到寫入指令時先不將資料寫入記憶體中,而是將 Cache 內儲存的該筆資料標示為已更改,等系統閒置時或該記憶體位址將被其他重複率更高的位址取代時,才真正將資料回寫
WriteThrough 的特點是效率較差,但安全性高,WriteBack 則正好相反反

Master Boot Record (MBR) v.s. GUID Partition Table (GPT)

http://www.petri.co.il/gpt-vs-mbr-based-disks.htm#

Multipath I/O

http://en.wikipedia.org/wiki/Multipath_I/O

Tagged Command Queuing (TCQ)

http://en.wikipedia.org/wiki/Tagged_Command_Queuing

Self-Monitoring, Analysis and Reporting Technology (S.M.A.R.T.)

http://en.wikipedia.org/wiki/S.M.A.R.T.

Fibre Channel

** Ports
=> A port in fibre channel terminology is any entity that actively communicates over the network, not necessarily a hardware port
=> http://en.wikipedia.org/wiki/Fibre_Channel#Ports

Topologies
=> FC_P2P : point to point
=> FC_AL : arbitrated loop
=> FC_SW : swiched fabric

AL_PA: Arbitrated Loop. Physical Address
=> Each device communicating on an arbitrated loop

World Wide Name (WWN)

=> Each device node has a WWNN
=> All ports share the WWNN, and own a unique WWPN
=> World Wide Node Name (WWNN)

=> World Wide Port Name (WWPN)

http://en.wikipedia.org/wiki/World_Wide_Name
http://ithelp.ithome.com.tw/question/10009960

Internet Storage Name Service (iSNS)

http://en.wikipedia.org/wiki/Internet_Storage_Name_Service

Clock synchronization

http://en.wikipedia.org/wiki/Clock_synchronization

SCSI command

http://en.wikipedia.org/wiki/SCSI_command

Bus, SAN, DAS, and NAS

SCSI (Small Computer System Interface)(並列SCSI)

SCSI initiator and target

Initiator : Host (computer), client
Target : Data storage device (HDD), server
Initiator and Target : mainly for communicating with each other
Target includes LUN (Logic Units) : mainly for execution command
Reference: http://en.wikipedia.org/wiki/SCSI_initiator_and_target

trminator :
Reference: http://en.wikipedia.org/wiki/SCSI_terminator

High voltage v.s. Low voltage

The difference between low voltage and high voltage for the same size cable is that cable size affects current carrying capacity, while power is voltage times current. As a result, the same size cable can transfer more power at high voltage than at low voltage. (Power = Voltage * Current, P = V * I)

This is why utility power transmission and distribution systems use high voltage. A typical transmission voltage in the US might be 69 kv, 138 kv, even 345 kv, and a typical distribution voltage in the US might by 13.2 kv.

Synchronous & Asynchronous Data transfer

Reference: http://www.ehow.com/facts_7463933_synchronous-asynchronous-data-transfer.html

SAS (Serial Attached SCSI)(串列SCSI)

由並列SCSI物理存儲介面演化而來。與並列方式相比，串列方式提供更快速的通訊傳輸速度以及更簡易的配置。此外SAS支援與串列ATA設備相容，且兩者可以使用相類似的電纜。SATA的硬碟可接在SAS的控制器使用，但SAS硬碟並不能接在SATA的控制器使用。

SAS是點對點（point-to-point）連接，並允許多個埠集中於單個控制器上，可以建立在主機板也可以另外添加。該技術建立在強大的並列SCSI通訊技術基礎上。SAS是採用SATA相容的電纜線採取點對點連接方式，從而在計算機系統中不需要建立菊花鏈（daisy-chaining）方式便可簡單地實現線纜安裝。

第一代SAS為陣列中的每個驅動器提供3.0 Gbps（約300 MBps）的傳輸速率。
第二代SAS為陣列中的每個驅動器提供6.0 Gbps（約600 MBps）的傳輸速率。

SAS由3種型別協定組成，根據連線的不同裝置使用相應的協定進行資料傳輸。

序列SCSI協定 (SSP) — 用於傳輸SCSI命令。
SATA通道協定 (STP) — 用於傳輸SATA資料。
SCSI管理協定 (SMP) — 用於對SAS裝置的維護和管理。

iSCSI (發音為 /аɪskʌzi/)(IP-SAN = IP-Storage Area Network)

與傳統的SCSI技術比較起來，iSCSI技術有以下三個革命性的變化：

把原來只用於本機的SCSI協同透過TCP/IP網路傳送，使連接距離可作無限的地域延伸。
連接的伺服器數量無限（原來的SCSI-3的上限是15）。
由於是伺服器架構，因此也可以實現在線擴容以至動態部署。

MPIO & MCS
http://blog.xuite.net/weirchen/blog/22184665-%E6%B7%B1%E5%85%A5%E4%BA%86%E8%A7%A3+iSCSI%E7%9A%84%E5%85%A9%E7%A8%AE%E5%A4%9A%E9%87%8D%E8%B7%AF%E5%BE%91%E5%AD%98%E5%8F%96%E6%A9%9F%E5%88%B6-2009.05.18%E5%A2%9E%E4%BF%AE

ATA (Advanced Technology Attachment)

(PATA = Parallel ATA)或稱(IDE = Integrated Drive Electronics)

SATA (Serial ATA)

是串列SCSI (SAS) 的孿生兄弟，兩者的排線相容，SATA硬碟可接上SAS介面。它是一種電腦匯流排，主要功能是用作主機板和大量儲存裝置（如硬碟及光碟機）之間的資料傳輸之用。SATA是已經完全取代舊式PATA（Parallel ATA或稱IDE）的新型硬碟介面，因採用序列方式傳輸資料而得名。現時，SATA分別有SATA 1.5Gbit/s、SATA 3Gbit/s和SATA 6Gbit/s三種規格。

主機板上的SATA 1.5Gb/s插口

頻寬速度換算

需要注意的是，在一些新的技術標準中，為了防止資料在高速傳輸中出錯而加入了校驗碼，比如PCI-E 2.0、USB 3.0和SATA 3.0中採用的是8/10編碼，每10位編碼中只有8位元是真實資料，這時單位換算就不再是1:8而是1:10了，USB 3.0的5Gbps速度實際上是理論500MB/s而非理論625MB/s，SATA 6Gbps的理論速度則是600MB/s而非750MB/s。所謂3Gb/s的演算法，3000MHz的頻率 x 每次發送一個資料 x 80%(8b/10b的編碼) / 8 bits per byte = 300Mbytes/s，同理1.5Gb/s也是這樣可算成150MB/s，也就是一般我們在買硬碟時，有時候會看到SATA 150MB/s / 300MB/s，有時候又會看到SATA 1.5Gb/s / 3Gb/s的緣故。

以USB 3.0而言，它擁有5Gbps的頻寬，每次發送一個資料 x 80%(8b/10b的編碼) / 8 bits per byte = 500Mbytes/s，所以USB 3.0的頻寬比SATA 3.0的600MB/s 還來的小。

SATA版本	頻寬	速度
SATA 3.0	6Gb/s	600MB/s
SATA 2.0	3Gb/s	300MB/s
SATA 1.0	1.5Gb/s	150MB/s

USB版本	頻寬	速度
USB 3.0	5Gb/s	500MB/s

Fibre Channel

http://en.wikipedia.org/wiki/Fibre_Channel

DAS

Refers to a digital storage system directly attached to a server or workstation, without a storage network in between.It is a retronym, mainly used to differentiate non-networked storage from SAN and NAS.

A typical DAS system is made of a data storage device (for example enclosures holding a number of hard disk drives) connected directly to a computer through a host bus adapter (HBA). Between those two points there is no network device (like hub, switch, or router), and this is the main characteristic of DAS.

The main protocols used for DAS connections are ATA, SATA, eSATA,^[1] SCSI, SAS, and Fibre Channel.

SAN

出於歷史原因，數據中心中最初都是SCSI 磁碟陣列的「孤島」群。每個單獨的小「島嶼」都是一個專門的直接連接存儲器應用，並且被視作無數個「虛擬硬碟驅動器」（例如LUNs）。本質上來說，SAN就是將一個個存儲「孤島」使用高速網路連接起來，這樣使得所有的應用可以訪問所有的磁碟。

作業系統會將SAN視為一組LUN，並且在LUN上維護自己的File System。這些不能在多個作業系統/主機之間進行共享的本地文件系統，具有非常高的可靠性和十分廣泛的應用。如果兩個獨立的本地文件系統存在於一個共享的LUN上，它們彼此沒有任何機制來知道對方的存在，沒有類似緩存同步(Distributed Lock Manager)的機制，所以可能發生數據丟失的情況。因此，在主機之間通過SAN共享數據，需要一些複雜的高級解決方案，例如SAN文件系統或者計算機集群。撇開這些問題，SAN對於提高存儲能力的應用有很大幫助，因為多個伺服器可以共享磁碟陣列上的存儲空間。SAN的一項典型應用是需要高速塊級別訪問的數據操作伺服器，比如電子郵件伺服器、資料庫、高利用率的文件伺服器等。

NAS

相對地，NAS允許多台計算機經過網路訪問同一個File system(Storage)，並且會自動同步它們的操作。由於NAS的引入使得SAN存儲可以被容易地轉換為NAS。

DAS、NAS 和 SAN 的比較

Daisy-chaining (菊花鏈)

Except for star-based networks, the easiest way to add more computers into a network is by daisy-chaining, or connecting each computer in series to the next. If a message is intended for a computer partway down the line, each system bounces it along in sequence until it reaches the destination. A daisy-chained network can take two basic forms: linear and ring.

In electrical and electronic engineering a daisy chain is a wiring scheme in which multiple devices are wired together in sequence or in a ring. Other than a full, single loop, systems which contain internal loops cannot be called daisy chains.

Linear Topology
Ring Topology

名稱	頻寬 (Mbit/s)	速度 (MB/s)	接線最大長度 (m)	電源供應	每頻道最多可接設備
eSATA	3,000	300	2 with eSATA HBA (1 with passive adapter)	否 ^[2]	1（15 with port multiplier）
串列SCSI	3,000	375	8	否	4
SATA 6.0Gb/s	6,000	600	1
SATA 3.0Gb/s	3,000	300	1	否	1（15 with port multiplier）
SATA 1.5Gb/s	1,500	150	1	否	1 per line
PATA 133	1,064	133	0.46（18 英寸）	否	2
FireWire 3200	3,144	393	100; alternate cables available for 100 m+	15 W, 12–25 V	63 (with hub)
FireWire 800	786	98.25	100^[3]	15 W, 12–25 V	63 (with hub)
FireWire 400	393	49.13	4.5^[3]^[4]	15 W, 12–25 V	63 (with hub)
USB 3.0	5,000	500		4.5 W, 5 V
USB 2.0	480	60	5^[5]	2.5 W, 5 V	127 (with hub)
USB 1.1	12	1.5
USB 1.0	1.5	0.1875
Ultra-640 SCSI	5,120	640	12	否	15 (plus the HBA)
Ultra-320 SCSI	2,560	320	12	否	15 (plus the HBA)
Fiber Channel 透過銅線	4,000	400	12	否	126 （16777216 with switches）
Fiber Channel 透過光纖	10,520	2000	2–50000	否	126 （16777216 with switches）
Infiniband 12X Quad-rate	120,000	12,000	5 （銅線）^[6]^[7] <10000 nbsp="" p="">	否	1 with Point to point Many with switched fabric

pip, virtualenv, virtualenvwrapper

pip

pip + virtualenv

virtualenvwrapper

Postgresql

Intall pip, virtualenv, virtualenvwrapper on Mac

相關文件

Storage Overview

Computer Data Storage

Parity bit

Single-ended signaling v.s. Differential signaling

SAMBA (SMB - Sever Message Block)

Scale out & Scale up

Storage Virtualization

Storage Performance Council (SPC-1)

Storage Service

Storage Architecture

RAID Levels

Physical Drive, Logical Drive, Logical Volume (Pool)

Cylinder, Head, Sector

Write Back Cache v.s. Write Through Cache (Write v.s. Read)

Master Boot Record (MBR) v.s. GUID Partition Table (GPT)

Multipath I/O

Tagged Command Queuing (TCQ)

Self-Monitoring, Analysis and Reporting Technology (S.M.A.R.T.)

Fibre Channel

World Wide Name (WWN)

Internet Storage Name Service (iSNS)

Clock synchronization

SCSI command

相關文件

Bus, SAN, DAS, and NAS

SCSI (Small Computer System Interface)(並列SCSI)

SCSI initiator and target

SAS (Serial Attached SCSI)(串列SCSI)

iSCSI (發音為 /аɪskʌzi/)(IP-SAN = IP-Storage Area Network)

ATA (Advanced Technology Attachment)

SATA (Serial ATA)

頻寬速度換算

Fibre Channel

DAS

SAN

NAS

Daisy-chaining (菊花鏈)

相關文件