HDFS Misc

HDFS configuration

Start here
Yahoo tutorial module 7

LZO compression

Good overview re LZO set up here

LZO compression notes (on Centos 6.1)

Install these packages:

lzo-2.03-3.1.el6.x86_64.rpm

lzop-1.02-0.9.rc1.el6.x86_64.rpm

Check:

$ rpm -qa | grep lzo

lzo-2.03-3.1.el6.x86_64

lzop-1.02-0.9.rc1.el6.x86_64

# Next unpack the tarball lzo-hadoop.tar.gz 

cd /mnt/nfs/vol1/packages

tar tvfz lzo-hadoop.tar.gz

drwx------ root/root         0 2012-04-12 11:30 native/

drwx------ root/root         0 2012-04-12 11:40 native/Linux-amd64-64/

drwx------ root/root         0 2012-04-12 11:40 native/Linux-amd64-64/lib/

-rwx------ root/root     67841 2012-04-12 11:40 native/Linux-amd64-64/lib/libgplcompression.so.0.0.0

lrwxrwxrwx root/root         0 2012-04-12 11:40 native/Linux-amd64-64/lib/libgplcompression.so.0 -> libgplcompression.so.0.0.0

lrwxrwxrwx root/root         0 2012-04-12 11:40 native/Linux-amd64-64/lib/libgplcompression.so -> libgplcompression.so.0.0.0

-rw------- root/root      1124 2012-04-12 11:40 native/Linux-amd64-64/lib/libgplcompression.la

-rw-r--r-- root/root    104238 2012-04-12 11:40 native/Linux-amd64-64/lib/libgplcompression.a

-rwx------ root/root    257940 2012-04-12 11:40 native/Linux-amd64-64/libtool

-rw------- root/root     18384 2012-04-12 11:40 native/Linux-amd64-64/Makefile

-rwx------ root/root     59681 2012-04-12 11:40 native/Linux-amd64-64/config.status

drwx------ root/root         0 2012-04-12 11:40 native/Linux-amd64-64/impl/

drwx------ root/root         0 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/

-rw------- root/root       332 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/LzoDecompressor.lo

-rw------- root/root     49048 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/LzoDecompressor.o

-rw------- root/root     54400 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/LzoCompressor.o

drwx------ root/root         0 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/.libs/

-rw------- root/root     49048 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/.libs/LzoDecompressor.o

-rw------- root/root     54400 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/.libs/LzoCompressor.o

drwx------ root/root         0 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/.deps/

-rw------- root/root         0 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/.deps/.dirstamp

-rw------- root/root      3803 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/.deps/LzoDecompressor.Plo

-rw------- root/root      3799 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/.deps/LzoCompressor.Plo

-rw------- root/root         0 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/.dirstamp

-rw------- root/root       326 2012-04-12 11:40 native/Linux-amd64-64/impl/lzo/LzoCompressor.lo

-rw------- root/root        28 2012-04-12 11:40 native/Linux-amd64-64/impl/stamp-h1

-rw------- root/root      4324 2012-04-12 11:40 native/Linux-amd64-64/impl/config.h

drwx------ root/root         0 2012-04-12 11:40 native/Linux-amd64-64/.libs/

-rwx------ root/root     67841 2012-04-12 11:40 native/Linux-amd64-64/.libs/libgplcompression.so.0.0.0

lrwxrwxrwx root/root         0 2012-04-12 11:40 native/Linux-amd64-64/.libs/libgplcompression.so.0 -> libgplcompression.so.0.0.0

lrwxrwxrwx root/root         0 2012-04-12 11:40 native/Linux-amd64-64/.libs/libgplcompression.so -> libgplcompression.so.0.0.0

lrwxrwxrwx root/root         0 2012-04-12 11:40 native/Linux-amd64-64/.libs/libgplcompression.la -> ../libgplcompression.la

-rw------- root/root      1124 2012-04-12 11:40 native/Linux-amd64-64/.libs/libgplcompression.lai

-rw------- root/root    104238 2012-04-12 11:40 native/Linux-amd64-64/.libs/libgplcompression.a

drwx------ root/root         0 2012-04-12 11:30 native/Linux-amd64-64/src/

drwx------ root/root         0 2012-04-12 11:30 native/Linux-amd64-64/src/com/

drwx------ root/root         0 2012-04-12 11:30 native/Linux-amd64-64/src/com/hadoop/

drwx------ root/root         0 2012-04-12 11:30 native/Linux-amd64-64/src/com/hadoop/compression/

drwx------ root/root         0 2012-04-12 11:38 native/Linux-amd64-64/src/com/hadoop/compression/lzo/

-rw------- root/root      1423 2012-04-12 11:40 native/Linux-amd64-64/src/com/hadoop/compression/lzo/com_hadoop_compression_lzo_LzoDecompressor.h

-rw------- root/root      1398 2012-04-12 11:40 native/Linux-amd64-64/src/com/hadoop/compression/lzo/com_hadoop_compression_lzo_LzoCompressor.h

-rw------- root/root      1123 2012-04-12 11:40 native/Linux-amd64-64/libgplcompression.la

-rw------- root/root     34625 2012-04-12 11:40 native/Linux-amd64-64/config.log

-rw------- root/root     62240 2012-04-12 11:40 hadoop-lzo-0.4.15.jar

cd ${HADOOP_HOME}/lib
tar xvfz lzo-hadoop.tar.gz
# Get the permissions right
chmod 640 hadoop-lzo-0.4.15.jar
cd native
chown –R hadoop:hadoop Linux-amd64-64
cd Linux-amd64-64
find . -type f –exec chmod 640 {} \;
find . -type d –exec chmod 750 {} \;

Note - this caught me out

cd ${HADOOP_HOME}/lib/native/Linux-amd64-64

-bash-4.1$ ln -s ./lib/libgplcompression.so.0.0.0 libgplcompression.so

-bash-4.1$ ln -s ./lib/libgplcompression.so.0.0.0 libgplcompression.so.0

-bash-4.1$ ls -atl libg*

lrwxrwxrwx 1 hadoop hadoop   32 Nov 30 13:43 libgplcompression.so.0 -> ./lib/libgplcompression.so.0.0.0

lrwxrwxrwx 1 hadoop hadoop   32 Nov 30 13:42 libgplcompression.so -> ./lib/libgplcompression.so.0.0.0

-rw-r----- 1 hadoop hadoop 1123 Apr 12  2012 libgplcompression.la

# Add the following to ${HADOOP_HOME}/conf/core-site.xml

io.compression.codecs

org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec,org.apache.hadoop.io.compress.BZip2Codec

added from build steps of lzo

io.compression.codec.lzo.class

com.hadoop.compression.lzo.LzoCodec

added from build steps of lzo