Introduction to HDFS and HDFS architecture: Hadoop mainly consists of two components: a distributed file system known as the Hadoop ...
Introduction to HDFS and HDFS architecture:
- Hadoop mainly consists of two components: a distributed file system known as the Hadoop distributed file system(HDFS) and a distributed processing framework named “MapReduce” (which is now supported by a component called “YARN (Resource Manager + task scheduler”)
- The HDFS is the storage component. Which provides a distributed architecture for exceeding large scale storage, which can easily be extended by scaling out.
HDFS has a master/slave architecture:
- 3 Master Demons
- Name Node: This daemon stores and maintains the metadata for HDFS.
- Secondary Name Node: Performs housekeeping functions for the Name Node.
- Job Tracker: Manages Map Reduce jobs, distributes individual tasks to machines running the Task Tracker.
- 2 Slave Demons
- Data Node: Stores actual HDFS data blocks.
- Task Tracker: Responsible for instantiating and monitoring individual Map and Reduce tasks.
 |
| HDFS has a master/slave architecture |
Data Storage in HDFS:
- Data storage in HDFS is different from the way files are stored in Windows or Linux.
- In HDFS, the system breaks down the file meant for storage into a set of Individual blocks and later stores the blocks in various slave nodes of the Hadoop cluster.
- Minimum size of block is 64MB and Maximum size can be integral (excluding zero and negative numbers) multiple of 64MB
Replicating data blocks :
- HDFS was designed to store the data even in inexpensive commodity hardware. The hardware can be sometimes unreliable. So to overcome any hardware failures, HDFS replicates data blocks that are stored in the system.
- Note: The norm is three copies of individual data blocks and any order can be stored in the data node. However, no two blocks of same type are stored in the data node (i.e. a1, a2 in the same data node 1).
From the above example the file (Sample.log) being divided into blocks of data (a,b,c,d,e,f). Which in turn are replicated in to three (a1,b1,c1,d1,e1,f1). (a2,b2,c2,d2,e2,f2) and (a3,b3,c3,d3,e3,f3)
 |
| DataNode 1 |
 |
| DataNode 2 |
 |
| DataNode 3 |
Name Node Memory concerns :
Secondary Name Node :
Anatomy of File Read:
Anatomy of File Write:
HDFS Commands:
The File System (FS) shell includes various shell-like commands that directly interact with the Hadoop Distributed File System (HDFS) as well as other file systems that Hadoop supports.
The fs shell can be invoked by $ bin/hdfs dfs <args>
Most of the commands in FS shell behave like corresponding Unix commands.
$cat Syntax: hdfs dfs -cat <<File path>>
Usage : Lists contents of the file.
$chgrp Syntax: hdfs dfs –chgrp <<group name>> <<file path>>
Usage : Lists contents of the file.
$chmod Syntax: hdfs dfs –chmod <<octal number>> <<file path>>
Usage: Change the permissions of files. With -R, make the change recursively through the directory structure. The user must be the owner of the file, or else a super-user.
$chown Syntax: hdfs dfs –chown <<Owner name>> <<file path>>
Usage: Change the owner of files. With -R, make the change recursively through the directory structure. The user must be a super-user.
$copyFromLocal Syntax: hdfs dfs –copyFromLocal
<<local input directory path>> << output hdfs directory path >>
Note: one can use put instead of copyFromLocal
Usage: Copies file from the local source to a output hdfs directory.
.$ copyToLocal Syntax: hdfs dfs –copyToLocal
<< input hdfs directory path >> <<local input directory path>>
Note: one can use get instead of copyToLocal
Usage: Copies file from the local hdfs directory to a output local directory.
$count Syntax: hdfs dfs –count <<directory path>>
Usage: Count the number of directories, files and bytes under the paths that match the specified file pattern.
$cp Syntax: hdfs dfs –cp <<source hdfs path>> <<output hdfs path>>
Usage: Copy files from source hdfs path to destination hdfs path.
$du Syntax: hdfs dfs -du <<directory path>>
Usage: Displays sizes of files and directories contained in the given directory or the length of a file in case its just a file.
$mkdir Syntax: hdfs dfs -mkdir <<directory name>>
Usage: Creates a directory
$ ls Syntax: hdfs dfs -ls <<directory or file>>
Usage:For a file or directory returns the statistics.
moveFromLocal Syntax: dfs -moveFromLocal <<local file path>> <<output hdfs path>>
Similar to put command, except that the source file is deleted after it's copied.
moveToLocal Syntax: hdfs dfs -moveToLocal <<input hdfs path>> <<output local file path>>
Displays a "Not implemented yet" message.
$mv Syntax: hdfs dfs -mv <<input file path>> <<output file path>>
Moves files from source to destination. This command allows multiple sources as well in which case the destination needs to be a directory. Moving files across file systems is not permitted.
$rm Syntax: hdfs dfs -rm <<File path>>
Delete the specified file
$setrep Syntax: hdfs dfs -setrep [-R] <<File path>>
Usage: Changes the replication factor of a file. -R option is for recursively increasing the replication factor of files within a directory.
$fsck Syntax: hdfs fsck –blocks <<file path>>
Usage: fsck – file system check
HDFS administrative commands
Hadoop in safe mode means :
- Name Node will be in safe mode and HDFS will be in read only mode
There are four important HDFS administrative commands
- $hdfs dfsadmin –safemode get
Usage: Gives the status of hdfs system. whether Hadoop is operating in safemode or not (i.e. either on or off state).
2) $hdfs dfsadmin –safemode enter
Usage: Hadoop operates in safemode (i.e on state)
3) $hdfs dfsadmin –safemode leave
Usage: Hadoop leaves safemode (i.e off state)
4) $hdfs dfsadmin –report
Usage: Gives report of the Hadoop system
COMMENTS