Sunday, October 27, 2013

Hadoop FileSystem (HDFS) Tutorial 1

In this tutorial I will show some common commands for HDFS operations.
If you don't have Hadoop setup in your linux, you can follow Hadoop Setup Guide

Log into Linux, "hduser" is the login used in following examples.

Start Hadoop If it's not running
$ start-dfs.sh
....
$ start-yarn.sh
Create someFile.txt in your home directory
hduser@ubuntu:~$ vi someFile.txt

Paste any text you want in to the file and save it.
Create Home Directory In HDFS (If it doesn't exist)
hduser@ubuntu:~$ hadoop fs -mkdir -p /user/hduser
Copy file someFile.txt from local disk to the user’s directory in HDFS.
hduser@ubuntu:~$ hadoop fs -copyFromLocal someFile.txt someFile.txt
Get a directory listing of the user’s home directory in HDFS
hduser@ubuntu:~$ hadoop fs –ls


Found 1 items
-rw-r--r--   1 hduser supergroup          5 2013-10-27 17:57 someFile.txt

Display the contents of the HDFS file /user/hduser/someFile.txt
hduser@ubuntu:~$ hadoop fs –cat /user/hduser/someFile.txt
Get a directory listing of the HDFS root directory
hduser@ubuntu:~$ hadoop fs –ls /
copy that file to the local disk, named as someFile2.txt
hduser@ubuntu:~$ hadoop fs –copyToLocal /user/hduser/someFile.txt someFile2.txt
Delete the file from hadoop hdfs
hduser@ubuntu:~$ hadoop fs –rm someFile.txt

Deleted someFile.txt


For a full list of commands, Please visit HDFS FileSystem Shell Commands. Please feel free to leave me any comments or suggestions.

59 comments:

  1. Note: if you get some warning saying "libhadoop.so.1.0.0 which might have disabled stack guard" it's because your system is 64 but the library is 32 try to rebuild the hadoop from source

    ReplyDelete
  2. Have a question on this step: Create Home Directory In HDFS (If it doesn't exist)

    What do you mean by Home Directory here? I have installed Hadoop, and created a dedicated user named hduser and the user has it's own home directory. Then whats the purpose of this Home Directory?

    Also, whats the root of my hdfs filesystem? Where exactly the file which I copied from my local directory is getting stored?

    Thanks very much for the intro though. I could get through them but was little confused about few things.

    Regards,
    Dnyanesh

    ReplyDelete
  3. Thanks for InformationHadoop Course will provide the basic concepts of MapReduce applications developed using Hadoop, including a close look at framework components, use of Hadoop for a variety of data analysis tasks, and numerous examples of Hadoop in action. This course will further examine related technologies such as Hive, Pig, and Apache Accumulo. HADOOP Online Training

    ReplyDelete
  4. Thanks for the valuable information.Recently I bought the hadoop videos from www.hadooponlinetutor.com for $20 only.Really the videos are too good.

    ReplyDelete
  5. Thank you so much for sharing this worthwhile to spent time on. You are running a really awesome blog. Keep up this good work Big Data Training

    ReplyDelete
  6. $20 can get you:

    a) Movie Tickets & popcorn,
    b) A cuppo for your car keys,
    c) A clothespin holder,
    d) A Hadoop 5-in-1 package as investment in your future.

    What are you buying today?
    Visit Now: http://bit.ly/1SqESgK

    ReplyDelete
  7. Thanks for sharing your informative article on Hive ODBC Driver. Your article is very descriptive and assists me to learn whole concept in detail. Hadoop Training in Chennai

    ReplyDelete
  8. The Hadoop tutorial you have explained is most useful for begineers who are taking Hadoop Administrator Online Training
    Thank you for sharing Such a good tutorials on Hadoop

    ReplyDelete
  9. There are lots of information about latest technology and how to get trained in them, like Best Hadoop Training In Chennai have spread around the web, but this is a unique one according to me. The strategy you have updated here will make me to get trained in future technologies Hadoop Training in Chennai By the way you are running a great blog. Thanks for sharing this blogs..

    ReplyDelete
  10. I found some useful information in your blog, it was awesome to read, thanks for sharing this great content to my vision, keep sharing..
    SalesForce Training in Chennai

    ReplyDelete
  11. Pretty article! I found some useful information in your blog, it was awesome to read,thanks for sharing this great content to my vision, keep sharing..
    Unix Training In Chennai

    ReplyDelete
  12. This information is impressive..I am inspired with your post writing style & how continuously you describe this topic. After reading your post,thanks for taking the time to discuss this, I feel happy about it and I love learning more about this topic..
    Android Training In Chennai In Chennai

    ReplyDelete
  13. SAP Training in Chennai
    This post is really nice and informative. The explanation given is really comprehensive and informative..

    ReplyDelete
  14. Oracle Training in chennai
    Thanks for sharing such a great information..Its really nice and informative..

    ReplyDelete
  15. Selenium Training in Chennai
    Wonderful blog.. Thanks for sharing informative blog.. its very useful to me..

    ReplyDelete
  16. Data warehousing Training in Chennai
    I am reading your post from the beginning, it was so interesting to read & I feel thanks to you for posting such a good blog, keep updates regularly..

    ReplyDelete
  17. Whatever we gathered information from the blogs, we should implement that in practically then only we can understand that exact thing clearly,
    but it’s no need to do it, because you have explained the concepts very well. It was crystal clear, keep sharing..
    Websphere Training in Chennai

    ReplyDelete
  18. Oracle DBA Training in Chennai
    Thanks for sharing this informative blog. I did Oracle DBA Certification in Greens Technology at Adyar. This is really useful for me to make a bright career..

    ReplyDelete
  19. This is really an awesome article. Thank you for sharing this.It is worth reading for everyone. Visit us:
    Oracle Training in Chennai

    ReplyDelete
  20. very nice blogs!!! i have to learning for lot of information for this sites...Sharing for wonderful information.Thanks for sharing this valuable information to our vision. You have posted a trust worthy blog keep sharing.
    Oracle DBA Training in Chennai

    ReplyDelete
  21. great article!!!!!This is very importent information for us.I like all content and information.I have read it.You know more about this please visit again.
    Oracle RAC Training in Chennai

    ReplyDelete
  22. Wonderful tips, very helpful well explained. Your post is definitely incredible. I will refer this to my friend.
    SalesForce Training in Chennai

    ReplyDelete
  23. I am reading your post from the beginning, it was so interesting to read & I feel thanks to you for posting such a good blog, keep updates regularly.
    Java Training in Chennai

    ReplyDelete
  24. Really awesome blog. Your blog is really useful for me. Thanks for sharing this informative blog. Keep update your blog.
    PHP Training in Chennai

    ReplyDelete
  25. Thanks for sharing this valuable information to our vision. You have posted a trust worthy blog keep sharing.Nice article i was really impressed by seeing this article, it was very interesting and it is very useful for me..
    Android Training in Chennai

    ReplyDelete
  26. Really awesome blog. Your blog is really useful for me. Thanks for sharing this informative blog. Keep update your blog.
    SAP Training in Chennai

    ReplyDelete
  27. Excellent information with unique content and it is very useful to know about the information based on blogs.
    Hadoop Training in Chennai

    ReplyDelete
  28. It is really very helpful for us and I have gathered some important information from this blog.If anyone wants to Selenium Training in Chennai reach Greens Technology training and placement academy.
    selenium Training in Chennai

    ReplyDelete
  29. We never miss a single post on this blog about hadoop. After attending hadoop online training, this site worked as a supplement to our technical knowledge about the subject related to cloud and other related platforms like hadoop.

    ReplyDelete
  30. Excellent information with unique content and it is very useful to know about the information based on blogs.
    Hadoop Training In Chennai | oracle apps financials Training In Chennai | advanced plsql Training In Chennai

    ReplyDelete
  31. Rajasthan Gram Panchayat 2252 Sathin Recruitment 2015-16

    Really appreciate you sharing this post.Much thanks again. Fantastic.............

    ReplyDelete
  32. Learning new technology would give oneself a true confidence in the current emerging Information Technology domain. With the knowledge of big data the most magnificent cloud computing technology one can go the peek of data processing. As there is a drastic improvement in this field everyone are showing much interest in pursuing this technology. Your content tells the same about evolving technology. Thanks for sharing this.

    Hadoop Training in Chennai | Hadoop Course in Chennai | Hadoop training institutes in chennai | Hadoop Training Chennai

    ReplyDelete
  33. Latest Govt JObs 2016


    Very efficiently written post. It will be valuable to anybody who employees it, including myself. Keep up the good work...................

    ReplyDelete
  34. There is a huge demand for professional big data analysts who are able to use the software which is used to process the big data in order to get accurate results. MNC's are looking for professionals who can process their data so that they can get into a accurate business decision which would eventually help them to earn more profits, they can serve their customers better, and their risk is lowered.
    big data training in chennai|big data training|big data course in chennai|big data training chennai|big data hadoop training in chennai

    ReplyDelete
  35. Some topics covered,may be it helps someone,HDFS is a Java-based file system that provides scalable and reliable data storage,
    and it was designed to span large clusters of commodity servers. HDFS has demonstrated production scalability of up to 200 PB of storage and a single cluster of 4500 servers, supporting close to a billion files and blocks.
    http://www.computaholics.in/2015/12/hdfs.html
    http://www.computaholics.in/2015/12/mapreduce.html
    http://www.computaholics.in/2015/11/hadoop-fs-commands.html

    ReplyDelete
  36. Really awesome blog. Your blog is really useful for me.
    Thanks for sharing this informative blog. Keep update your blog.
    Oracle Training In Chennai

    ReplyDelete
  37. Best SAS Training Institute In Chennai It’s too informative blog and I am getting conglomerations of info’s about Oracle interview questions and answer .Thanks for sharing, I would like to see your updates regularly so keep blogging.

    ReplyDelete
  38. Salesforce.com is an american company which offfers CRM based cloud services and it is loved globally for it quality services
    salesforce training in chennai|salesforce training institute in chennai | salesforce course in chennai

    ReplyDelete
  39. SAS stands for statistical analysis system which is a analysis tool developed by SAS institute and with the help of this tool data driven decisions can be taken which is helpful for the bsuiness.
    SAS training in Chennai | SAS course in Chennai | SAS training institute in Chennai

    ReplyDelete
  40. Kentucky Derby 2016 live

    http://kentuckyderby2016live.co/
    http://kentuckyderbylive.xyz/

    http://kentuckyderby2016live.co/
    http://kentuckyderbylive.xyz/

    ReplyDelete
  41. Thanks for this post is very informative and interesting.all the points are very useful. Simple but very effective writing. Thanks for sharing such a nice post.

    Hadoop Training in Chennai

    ReplyDelete
  42. Amazing content.If you are interested instudying nodejs visit this website. Nodejs is an open source, server side web application that enables you to build fast and scalable web application that is capable of running large number of simultaneous connections that has high throughput.
    Node js Training in Chennai | Node JS training institute in chennai

    ReplyDelete
  43. Indy 500 live

    http://watchindy500live.xyz/
    http://indy-500-live.xyz/
    http://indianapolis500-live.xyz/
    http://cocacola600live.xyz/
    http://stateoforigin2016.com/
    http://nswvsqldlive.xyz/
    http://queenslandvsnswlive.xyz/

    http://watchindy500live.xyz/
    http://indy-500-live.xyz/
    http://indianapolis500-live.xyz/
    http://cocacola600live.xyz/
    http://stateoforigin2016.com/
    http://nswvsqldlive.xyz/
    http://queenslandvsnswlive.xyz/

    ReplyDelete



  44. I have been following you for a couple of months now but this is my first time commenting on a blog post. Thank you for sharing your knowledge and experience with us. Keep up the good work. Already bookmarked for future reference.

    SAP training in Chennai

    ReplyDelete
  45. I agree with your thoughts!!! As the demand of hadoop programming application keeps on increasing, there is massive demand for hadoop professionals in software development industries.
    Regards,
    Bigdata Hadoop Training in Chennai | Hadoop course in Chennai

    ReplyDelete
  46. http://2016usopenlive.xyz/
    http://usopenchampionship2016.blogspot.com/
    http://usopenchampionship2016live.hatenablog.com/
    http://usopenchampionship2016live.kinja.com/
    http://usopenchampionship2016livestream.over-blog.com/
    http://openchampionship2016livestram.tumblr.com/
    https://sites.google.com/site/usopenchampionship2016live/
    http://usopenchampionship2016liv.nation2.com/
    https://usopenchampionship2016live.wordpress.com/
    http://usopenchampionship2016live.weebly.com/
    http://usopenchampionship2016live.jimdo.com/
    http://usopenchampionship2016live.myfreesites.net/
    http://usopenlive.webnode.com/
    http://usopenchampionship2016live.webs.com/
    http://usopenchampionship2016live.yolasite.com/
    http://www.simplesite.com/builder/pages/preview3.aspx

    ReplyDelete
  47. http://usavsargentina.xyz/
    http://usavsargentinalive.tumblr.com/
    http://usavsargentinalive.over-blog.com/
    https://usavsargentinalive.wordpress.com/
    https://sites.google.com/site/usavsargentinalive/
    http://usavsargentinalive.blogspot.com/
    https://www.pinterest.com/usavsargentina/
    http://usavsargentinalive.jimdo.com/
    http://usavsargentinalive.nation2.com/
    http://usavsargentinalive.kinja.com/
    http://usavsargentinalive.sportsblog.com/
    http://usavsargentinalive.full-design.com/
    http://usavsargentinalive.total-blog.com/
    http://usavsargentinalive.blogzet.com/
    http://usavsargentinalive.blogminds.com/
    http://usavsargentinalive.suomiblog.com/
    http://usavsargentinalive.isblog.net/
    http://usavsargentinalive.shotblogs.com/
    http://usavsargentinalive.canariblogs.com/
    http://usavsargentina.blogdon.net/
    http://usavsargentinalive.mybjjblog.com/
    http://usavsargentina.blogdigy.com/
    http://usavsargentina.blogdon.net/
    http://usavsargentinalive.blogkoo.com/
    http://usavsargentina.blogdigy.com/
    http://usavsargentinalive.touchblogs.com/
    http://peoplespace.com/newsfeed/1279?ft=site

    ReplyDelete
  48. Great and useful article. Creating content regularly is very tough. Your points are motivated me to move on.


    SEO Company in Chennai

    ReplyDelete
  49. http://theopenchampionshiplive.xyz/
    http://usopenchampionshiplive.xyz/watch-free/

    ReplyDelete
  50. http://nflfootball-live.com/
    http://rio2016live.net/

    ReplyDelete
  51. http://watcholympic2016.net/
    http://www.rio2016live.net/
    http://watcholympic2016.net/olympic-games-rio-2016-live/
    http://watcholympic2016.net/rio-olympics-2016-live-online/
    http://rio2016live.net/tag/rio-2016-live/
    https://www.facebook.com/Rio-Olympics-2016-Free-Live-Online-1309804505704053
    https://twitter.com/rioolympicfree
    https://www.pinterest.com/Olympics2016hd/
    https://www.instagram.com/watcholympicslive/
    https://vk.com/rioolympics2016live
    http://watchrioolympicslive.tumblr.com/
    http://rioolympic2016livestream.over-blog.com/
    http://rioolympics-live.blogspot.com/
    http://watcholympicslive.kinja.com/
    http://watcholympics.hatenablog.com/
    http://watcholympics2016live.blogdon.net/
    http://rio2016live.myblog.de/
    http://rio2016live.hazblog.com/
    http://profile.hatena.ne.jp/rio2016live/
    http://rio2016live.hatenadiary.com/

    ReplyDelete
  52. The world is changing fast. people are also being transformed. day by day we are becoming more dependant on degital system. you are making me think of this really. You have a great method of sharing your thoughts Ace Maxs, Cara Pemesanan Cordyceps, Cara Pemesanan Cordyceps, Cara Pemesanan Slimming Capsule, Jelly Gamat QnC.

    ReplyDelete
  53. I have read your blog its very attractive and impressive. I like it your blog.

    Java Training in Chennai Core Java Training in Chennai Core Java Training in Chennai

    ReplyDelete
  54. Java Online Training Java Online Training Core Java 8 Training in Chennai Core java 8 online training JavaEE Training in Chennai Java EE Training in Chennai

    ReplyDelete
  55. These provided information was really so nice,thanks for giving that post and the more skills to develop after refer that post. Your articles really impressed for me,because of all information so nice.

    SEO Company in India

    ReplyDelete