Godhand’s Blog

CAP theorem

2022-10-20T02:56:00+00:00

CAP

網路分區容錯(Tolerance towards Network Partition)
一致性(Consistency)
可用性(Availability)

三項最多取兩項 (實務上並非全有全無，而是傾向)

CA without P: 系統在網絡分區的情況下，存在未定義的行為。 AP without C: 不保證數據一致性。(使用樂觀鎖，並處理不一致所造成的conflict) CP without A: 數據只能在保證一致性的情況下才能被使用。(使用悲觀鎖等待所有節點一致)

近似(但仍可以在不同層級交疊使用)

CA: replica (一份資料複製多份放在各個節點)

在網路分區的情況下，各節點仍然可用，但一致性是未定義的) CP: sharding (特定資料只放在特定節點)
節點壞掉時部分資料不可用 AP: 綜合

References

Brewer’s CAP Theorem

Backup MongoDB in Docker

2022-08-19T02:56:00+00:00

Backup

docker exec CONTAINER mongodump -u USER -p PASS --authenticationDatabase AUTH_DB --gzip --archive > backup.gzip

Restore

cat backup.gzip | docker exec -i CONTAINER mongorestore -u USER -p PASS --authenticationDatabase AUTH_DB --gzip --archive [--nsInclude DATABASE.COLLECTION]

Bit manipulation

2022-06-14T10:48:44+00:00

power of 2
- !(x & (x-1)
least significant 1 bit
- x & -x
bitset
- add: bitset |= 1 << x
- has: (bitset >> x) & 1
- remove: bitset &= ~(1 << x)
- union: bitsetA | bitsetB
- intersection: bitsetA & bitsetB
- difference: bitsetA - (bitsetA & bitsetB)
- iterate non-empty subset (reversed): for(let subset = bitset; subset > 0; subset = (subset - 1) & bitset) { ... }

Split file by size

2021-01-29T08:05:03+00:00

split -b 10M -d --additional-suffix=.part foo bar
cat bar*.part > foo

b: split every N bytes
a: generate suffixes of length N (default 2)
d: use numeric suffixes
additional-suffix=SUFFIX: append additional static SUFFIX

Forward proxy and reverse proxy via ssh

2021-01-15T02:00:00+00:00

# forward proxy
#   request -> LOCAL -> SSH_SERVER -> TARGET
ssh -NfL $LOCAL_PORT:$TARGET:$TARGET_PORT $USER@$SSH_SERVER

# reverse proxy
#   request -> SSH_SERVER -> LOCAL -> TARGET
ssh -NfR $SSH_SERVER_PORT:$TARGET:$TARGET_PORT $USER@$SSH_SERVER

N: Do not execute a remote command. This is useful for just forwarding ports.
f: background
L: Specifies that connections to the given TCP port or Unix socket on the local (client) host are to be forwarded to the given host and port, or Unix socket, on the remote side.
R: Specifies that connections to the given TCP port or Unix socket on the remote (server) host are to be forwarded to the local side.

install packages offline

2020-09-09T08:05:03+00:00

centos with yum

yum --downloadonly --downloaddir=`pwd` install vim
tar zcvf vim.tgz *.rpm
# copy to offline machine
tar zxvf vim.tgz
yum install *.rpm

ubuntu with apt

apt depends --recurse -i vim | grep '^\w' | sort -u | xargs apt download
tar zcvf vim.tgz *.deb
# copy to offline machine
tar zxvf vim.tgz
dpkg -i *.deb

UTF (Unicode Transformation Format)

2020-07-28T08:05:03+00:00

BOM

Byte Order Mark
位於檔案最前面(相當於檔案的header)
用來暗示這串 bytes 是使用 utf 編碼
用來區別 big-endian 與 little-endian

utf8

每個字符佔據 1~4 bytes, 可表示所有字符
每個字符的第 frist byte 可以得知此字符總共佔據多少個 byte
- 1 byte 字符 (U+00000 ~ U+00007F): 0xxxxxxx (其實就是ascii)
- 2 bytes 字符 (U+00080 ~ U+0007FF): 110xxxxx 10xxxxxx
- 3 bytes 字符 (U+00800 ~ U+00FFFF): 1110xxxx 10xxxxxx 10xxxxxx
- 4 bytes 字符 (U+10000 ~ U+10FFFF): 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
utf8 的 BOM 為 0xEF 0xBB 0xBF
utf8 解碼時, 一次只解讀 1 byte, 所以並不會受 Endianness 所影響, 所以不需 BOM, 就可以正確解讀
ascii 無須而外處理即可當作 utf8 使用, 而 utf8 只有在字符皆為 ascii 內, 且檔首無 BOM, 才可以轉成ascii
計算字串總長度, 或是取第N個字符, 因為不等長的特性所以都需從頭走訪一遍

uft16

每個字符佔據 2 bytes 或 4 bytes, 可表示所有字符
- 2 bytes 字符 (U+000000 ~ U+00D7FF, U+00E000 ~ U+00FFFF): xxxxxxxx xxxxxxxx
- 4 bytes 字符 (U+010000 ~ U+10FFFF): 110110xx xxxxxxxx 110111xx xxxxxxxx
utf16le 的 BOM 為 0xFF 0xFE
utf16be 的 BOM 為 0xFE 0xFF
utf16 解讀時, 即使用錯 Endianness 也都會找到對應的字符, 所以必須有 BOM, 才能確保正確解讀
計算字串總長度, 或是取第N個字符, 因為不等長的特性所以都需從頭走訪一遍

uft32

每個字符都固定佔據 4 bytes, 可表示所有字符
utf32le 的 BOM 為 0xFF 0xFE 0x00 0x00
utf32be 的 BOM 為 0x00 0x00 0xFE 0xFF
utf32 解讀時, 即使用錯 Endianness 也都會找到對應的字符, 所以必須有 BOM, 才能確保正確解讀

CLI

使用 printf 加入 BOM

  $ (printf '\xEF\xBB\xBF'; $UTF8) > $UTF8_BOM
  $ (printf '\xFF\xFE'; cat $UTF16LE) > $UTF16LE_BOM
  $ (printf '\xFE\xFF'; cat $UTF16BE) > $UTF16BE_BOM
  $ (printf '\xFF\xFE\x00\x00'; cat $UTF32LE) > $UTF32LE_BOM
  $ (printf '\x00\x00\xFE\xFF'; cat $UTF32BE) > $UTF32BE_BOM

使用 dd 移除 BOM

  $ dd bs=3 skip=1 if=$UTF8_BOM of=$UTF8 &>/dev/null
  $ dd bs=2 skip=1 if=$UTF16LE_BOM of=$UTF16LE &>/dev/null
  $ dd bs=4 skip=1 if=$UTF32BE_BOM of=$UTF32BE &>/dev/null

使用 dd 轉換 Endianness

  $ dd conv=swab if=$UTF16LE of=$UTF16BE &>/dev/null
  $ dd conv=swab if=$UTF16LE_BOM of=$UTF16BE_BOM &>/dev/null

使用 iconv 轉換編碼

在使用 iconv 在 utf編碼間轉換編碼時需注意

以 utf16 跟 utf32 作為輸出格式時, 會自動幫你在輸出加上 BOM

以 utf16 跟 utf32 作為輸入格式時, 會自動幫你去掉輸入的 BOM

  $ iconv -f utf8 -t utf16 $UTF8 > $UTF16LE_BOM
  $ iconv -f utf8 -t utf16 $UTF8_BOM > $UTF16LE_BOM_BOM
  $ iconv -f utf8 -t utf16le $UTF8 > $UTF16LE
  $ iconv -f utf8 -t utf16le $UTF8_BOM > $UTF16LE_BOM
  $ iconv -f utf16 -t utf8 $UTF16LE_BOM > $UTF8
  $ iconv -f utf16le -t utf8 $UTF16LE_BOM > $UTF8_BOM

set up MSYS2 on windows

2020-04-16T08:05:03+00:00

install MSYS2

Go to msys2.org and download msys2-x86_64-20190524.exe
After installed, MSYS2 is avaliable at C:\msys64\usr\bin\mintty.exe

post-install MSYS2

first, update system at the first time start up
```
pacman -Syu
```
after finish, close(NOT Ctrl+c) the window and reopen again
update again
```
pacman -Sy
pacman -Syu
```
(optional) add host environment variable C:\msys64\mingw64\bin to PATH

install toolchain in MSYS2

pacman -S mingw-w64-x86_64-toolchain make

shadow gcc

echo "alias gcc=\"x86_64-w64-mingw32-gcc\"" >> ~/.bashrc

Backup MySQL in Docker

2020-04-15T08:05:03+00:00

Backup

docker exec CONTAINER /usr/bin/mysqldump -u USER --password=PASS DATABASE > backup.sql

Restore

cat backup.sql | docker exec -i CONTAINER /usr/bin/mysqldump -u USER --password=PASS DATABASE

Transducer

2019-10-25T09:36:00+00:00

const R = require('ramda')
const { filter, reduce, map } = R
const { compose, allPass } = R
const { inc, length, includes, add } = R
const even = n => n % 2 == 0

var arr = ["cat", "bird", "pig", "dog", "fish"]
// We know that the function in mutiple map,
// we can compose to decrease the iteration.
console.log(
    map(inc, map(length, arr)),
    map(compose(inc, length))(arr)
)
// Function in mutiple filter can compose in another way
console.log(
    filter(includes('g'), filter(includes('i'), arr)),
    filter(allPass([includes('g'), includes('i')]))(arr)
)
// But it seem not works well between map and filter.
console.log(map(inc, filter(even, map(length, arr))))
console.log("something like compose(map(inc), filter(even), map(length))(arr)")

// Transducers is just composition between map and filter.

// Let's take a look at map and filter. (NOT real implementation)
// map = (f, arr) => arr.map(f)
// map :: (a -> b) -> [a] -> [b]
// f :: a -> b
//
// filter = (pred, arr) => arr.filter(pred)
// filter :: (a -> Bool) -> [a] -> [a]
// pred :: a -> Bool

// The signature is so different, how can I compose them?

// First, we know we can use reduce to implement map and filter.
// reduce = (reducer, initVal, arr) => arr.reduce(reducer, initVal)
// reduce :: (a -> b -> a) -> a -> [b] -> a
// recucer :: a -> b -> a 
const append = (arr, e) => [...arr, e]
// append :: [a] -> a -> [a]
console.log(
    map(length, arr),
    reduce((a, c) => append(a, length(c)), [], arr)
)
console.log(
    filter(includes('i'), arr),
    reduce((a, c) => includes('i')(c) ? append(a, c) : a, [], arr)
)
// Let's create some High-Order-Functions to create reducer,
// and pass it to reduce.
const mapReducer = f => (a, c) => append(a, f(c))
// mapReducer :: f -> reducer
// mapReducer :: (a -> b) -> ([b] -> [a] -> [b])
const filterReducer = pred => (a, c) => pred(c) ? append(a, c) : a
// filterReducer :: pred -> reducer
// filterReducer :: (a -> Bool) -> ([a] -> a -> [a])

// The reduce-version works well.
console.log(
    map(length, arr),
    reduce(mapReducer(length), [], arr),
)
console.log(
    filter(includes('i'), arr),
    reduce(filterReducer(includes('i')), [], arr)
)

// Surprisingly, "append" is also a reducer!!
// const append = (arr, e) => [...arr, e]
// append :: reducer
// append :: [a] -> a -> [a]
// We can extract 'append'
const mapping = f => r => (a, c) => r(a, f(c))
// mapping :: f -> reducer -> reducer'
// mapping :: (a -> b) -> (c -> b -> c) -> (c -> a -> b)
const filtering = pred => r => (a, c) => pred(c) ? r(a, c) : a
// filtering :: pred -> reducer -> reducer'
// filtering :: (a -> Bool) -> (b -> a -> b) -> (b -> a -> b)

// Still works as normal.
console.log(
    map(length, arr),
    reduce(mapping(length)(append), [], arr)
)
console.log(
    filter(includes('i'), arr),
    reduce(filtering(includes('i'))(append), [], arr)
)

// In this version, while mapping and filtering are partial applied,
// they become (reducer -> reducer') which is composable.

// Notice the order 
const xform = compose(mapping(length), filtering(even), mapping(inc))
// xform :: reducer -> reducer'
console.log(
    map(inc, filter(even, map(length, arr))),
    reduce(xform(append), [], arr)
)

// To reduce a large array, transducer is your friend.
const bigArr = new Array(100000).fill(1).map(() => Math.round(Math.random() * 10))
const xform2 = compose(filtering(even), mapping(inc))
console.time("normal")
console.log(reduce(xform2(add), 0, bigArr))
console.timeEnd("normal")

console.time("transducer")
console.log(reduce(add, 0, map(inc, filter(even, bigArr))))