HDFS Misc

HDFS configuration

Start here
Yahoo tutorial module 7

LZO compression

Good overview re LZO set up here

LZO compression notes (on Centos 6.1)

Install these packages:

lzo-2.03-3.1.el6.x86_64.rpm

lzop-1.02-0.9.rc1.el6.x86_64.rpm

Check:

$ rpm -qa | grep lzo

lzo-2.03-3.1.el6.x86_64

lzop-1.02-0.9.rc1.el6.x86_64

# Next unpack the tarball lzo-hadoop.tar.gz 

cd /mnt/nfs/vol1/packages

tar tvfz lzo-hadoop.tar.gz

drwx------ root/root         0 2012-04-12 11:30 native/

drwx------ root/root         0 2012-04-12 11:40 native/Linux-amd64-64/

drwx------ root/root         0 2012-04-12 11:40 native/Linux-amd64-64/lib/

-rwx------ root/root     67841 2012-04-12 11:40 native/Linux-amd64-64/lib/libgplcompression.so.0.0.0

lrwxrwxrwx root/root         0 2012-04-12 11:40 native/Linux-amd64-64/lib/libgplcompression.so.0 -> libgplcompression.so.0.0.0

lrwxrwxrwx root/root         0 2012-04-12 11:40 native/Linux-amd64-64/lib/libgplcompression.so -> libgplcompression.so.0.0.0

-rw------- root/root      1124 2012-04-12 11:40 native/Linux-amd64-64/lib/libgplcompression.la

-rw-r--r-- root/root    104238 2012-04-12 11:40 native/Linux-amd64-64/lib/libgplcompression.a

-rwx------ root/root    257940 2012-04-12 11:40 native/Linux-amd64-64/libtool

-rw------- root/root     18384 2012-04-12 11:40 native/Linux-amd64-64/Makefile

-rwx------ root/root     59681 2012-04-12 11:40 native/Linux-amd64-64/config.status

drwx------ root/root         0 2012-04-12 11:40 native/Linux-amd64-64/impl/

drwx------ root/root         0 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/

-rw------- root/root       332 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/LzoDecompressor.lo

-rw------- root/root     49048 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/LzoDecompressor.o

-rw------- root/root     54400 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/LzoCompressor.o

drwx------ root/root         0 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/.libs/

-rw------- root/root     49048 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/.libs/LzoDecompressor.o

-rw------- root/root     54400 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/.libs/LzoCompressor.o

drwx------ root/root         0 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/.deps/

-rw------- root/root         0 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/.deps/.dirstamp

-rw------- root/root      3803 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/.deps/LzoDecompressor.Plo

-rw------- root/root      3799 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/.deps/LzoCompressor.Plo

-rw------- root/root         0 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/.dirstamp

-rw------- root/root       326 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/LzoCompressor.lo

-rw------- root/root        28 2012-04-12 11:40 native/Linux-amd64-64/impl/stamp-h1

-rw------- root/root      4324 2012-04-12 11:40 native/Linux-amd64-64/impl/config.h

drwx------ root/root         0 2012-04-12 11:40 native/Linux-amd64-64/.libs/

-rwx------ root/root     67841 2012-04-12 11:40 native/Linux-amd64-64/.libs/libgplcompression.so.0.0.0

lrwxrwxrwx root/root         0 2012-04-12 11:40 native/Linux-amd64-64/.libs/libgplcompression.so.0 -> libgplcompression.so.0.0.0

lrwxrwxrwx root/root         0 2012-04-12 11:40 native/Linux-amd64-64/.libs/libgplcompression.so -> libgplcompression.so.0.0.0

lrwxrwxrwx root/root         0 2012-04-12 11:40 native/Linux-amd64-64/.libs/libgplcompression.la -> ../libgplcompression.la

-rw------- root/root      1124 2012-04-12 11:40 native/Linux-amd64-64/.libs/libgplcompression.lai

-rw------- root/root    104238 2012-04-12 11:40 native/Linux-amd64-64/.libs/libgplcompression.a

drwx------ root/root         0 2012-04-12 11:30 native/Linux-amd64-64/src/

drwx------ root/root         0 2012-04-12 11:30 native/Linux-amd64-64/src/com/

drwx------ root/root         0 2012-04-12 11:30 native/Linux-amd64-64/src/com/hadoop/

drwx------ root/root         0 2012-04-12 11:30 native/Linux-amd64-64/src/com/hadoop/compression/

drwx------ root/root         0 2012-04-12 11:38 native/Linux-amd64-64/src/com/hadoop/compression/lzo/

-rw------- root/root      1423 2012-04-12 11:40 native/Linux-amd64-64/src/com/hadoop/compression/lzo/com_hadoop_compression_lzo_LzoDecompressor.h

-rw------- root/root      1398 2012-04-12 11:40 native/Linux-amd64-64/src/com/hadoop/compression/lzo/com_hadoop_compression_lzo_LzoCompressor.h

-rw------- root/root      1123 2012-04-12 11:40 native/Linux-amd64-64/libgplcompression.la

-rw------- root/root     34625 2012-04-12 11:40 native/Linux-amd64-64/config.log

-rw------- root/root     62240 2012-04-12 11:40 hadoop-lzo-0.4.15.jar

cd ${HADOOP_HOME}/lib
tar xvfz lzo-hadoop.tar.gz
# Get the permissions right
chmod 640 hadoop-lzo-0.4.15.jar
cd native
chown –R hadoop:hadoop Linux-amd64-64
cd Linux-amd64-64
find . -type f –exec chmod 640 {} \;
find . -type d –exec chmod 750 {} \;

Note - this caught me out

cd ${HADOOP_HOME}/lib/native/Linux-amd64-64

-bash-4.1$ ln -s ./lib/libgplcompression.so.0.0.0 libgplcompression.so

-bash-4.1$ ln -s ./lib/libgplcompression.so.0.0.0 libgplcompression.so.0

-bash-4.1$ ls -atl libg*

lrwxrwxrwx 1 hadoop hadoop   32 Nov 30 13:43 libgplcompression.so.0 -> ./lib/libgplcompression.so.0.0.0

lrwxrwxrwx 1 hadoop hadoop   32 Nov 30 13:42 libgplcompression.so -> ./lib/libgplcompression.so.0.0.0

-rw-r----- 1 hadoop hadoop 1123 Apr 12  2012 libgplcompression.la

# Add the following to ${HADOOP_HOME}/conf/core-site.xml

io.compression.codecs

org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec,org.apache.hadoop.io.compress.BZip2Codec

added from build steps of lzo

io.compression.codec.lzo.class

com.hadoop.compression.lzo.LzoCodec

added from build steps of lzo

Unix stuff - fsck usb h/d eg

Unix bits

fsck external hard disk (/mnt/usb) that lost power


# df -h


/dev/sdh1             1.8T  1.7T   57G  97% /mnt/usb2
/dev/sdf1             1.8T  196M  1.7T   1% /mnt/usb

# cd /mnt/usb
# ls 
ls: reading directory .: Input/output error
# fdisk -l
# fdisk -l | grep dev


Disk /dev/mapper/RootVolume-OptVol: 34.4 GB, 34359738368 bytes
Disk /dev/sdh: 2000.4 GB, 2000398934016 bytes
/dev/sdh1               1      243202  1953514583+  ee  GPT
Disk /dev/sdf: 2000.4 GB, 2000398934016 bytes
/dev/sdf1               1      243202  1953514583+  ee  GPT

# add device to fstab
# vi /etc/fstab
# cat /etc/fstab


/dev/sdh1             1.8T  1.7T   57G  97% /mnt/usb2
/dev/sdf1             1.8T  196M  1.7T   1% /mnt/usb

# /mnt/usb
# df -h


/dev/sdh1             1.8T  1.7T   57G  97% /mnt/usb2
/dev/sdf1             1.8T  196M  1.7T   1% /mnt/usb

# fsck.ext3 /dev/sdf1

e2fsck 1.41.12 (17-May-2010)
/dev/sdf1 is mounted.  

WARNING!!!  The filesystem is mounted.   If you continue you ***WILL***
cause ***SEVERE*** filesystem damage.

Do you really want to continue (y/n)? yes

/dev/sdf1: recovering journal
/dev/sdf1: clean, 11/122101760 files, 7713452/488378637 blocks

Throttling traffic via iptables

Useful link answer to: Throttle Traffic via iptables question

Note from Graham Hargreaves (prob based on someone else) ...

Not very accurate but definitely restricts the flow.

Change modemif variable to be the interface you need to throttle.

The example below never allowed a download to go above 130kbs and set to ambit it never went above 3Mbs (Probably needs some more testing)

To turn on:

#!/bin/bash

modemif=eth0

iptables -t mangle -A POSTROUTING -o $modemif -p tcp -m tos --tos Minimize-Delay -j CLASSIFY --set-class 1:10

iptables -t mangle -A POSTROUTING -o $modemif -p tcp --dport 80 -j CLASSIFY --set-class 1:10

iptables -t mangle -A POSTROUTING -o $modemif -p tcp --dport 443 -j CLASSIFY --set-class 1:10

tc qdisc add dev $modemif root handle 1: htb default 12

tc class add dev $modemif parent 1:1 classid 1:12 htb rate 50kbit ceil 50kbit

To turn the above off simply run:

tc qdisc del dev eth0 root

iptables