image classification using cnn

 Various types of data compression in MapReduce

When Hadoop word comes to mind instantly, one more word also comes side by side in mind which is big data. Big data means a very large amount of data. When we need to play with a large amount of data there will always be an issue of scarcity of space. So, how can we or Hadoop as architecture can handle such a critical issue? Hadoop has provided very nice and important to rescue us from this issue. The resolution is data compression. We can do data compression using different Hadoop libraries on our huge dataset. If you are still not clear about what are the benefits of data compression in Hadoop let me show you. As we will compress the dataset size required to store data will decrease drastically. On the other end as we all know we need to transfer data among the Hadoop clusters from one machine to another. So, as a result of data compression data size will decrease, and eventually, the speed at which data will be transferred over the network will also increase. As we understood now the purpose of data compression in Map Reduce programming let’s see how the different types of compression options available in Hadoop map-reduce.

Types of Compression in Map Reduce

There are different types of compression techniques available. Each has different characteristics. Let’s see a list of available compression techniques and understand each one of them in detail.

·         Deflate

·         Gzip

·         Bzip2

·         LZO

·         LZ4

·         Snappy

Here we can distinguish each one of them based on the trade-off between space and time. It means if the speed of compression is more then compression will be less and if the time required to compress will be more then the compression will be of small space.

Deflate

Deflate is a lossless data compression file format that uses a combination of LZ77 and Huffman coding. Its file extension is .deflate. Having single file and not splittable. It stores data as a series of blocks.

Gzip

Gzip is a compression technique of type deflate. It is being used to store compressed data. Gzip has a .gz extension and has gzip tool that works as a single file and is not splittable. It provides a high compression ratio so the time required to compress files will also be high and it will take more time. It also uses high resource utilization for compression and decompression.

Zip

Zip is having a tool zip to perform compression. It uses deflate compression algorithm internally. It has a zip extension. It can have multiple files and is splittable at the file boundaries level.

Bzip2

Bzip2 is fairly slow comparatively. But, it has more compression ratio. It can’t have multiple files and but it is splittable. It uses the bzip2 algorithm and uses the bzip2 tool. It takes more time to compress and decompress data.

LZO

LZO is another compression technique. It uses lzop tool to perform compression. It uses the LZO algorithm internally. It can not have multiple files and is not splittable. It is a faster compression technique. It uses a low compression ratio.

LZ4

LZ4 is a compression technique that can be used at any speed-to-compression ratio. We don’t need external indexing in this technique. We can split the compressed files in this approach.

Snappy

Snappy is a technique that provides the best trade-off between speed and compression. It focuses on more speed and less compression. It is used widely in corporate organizations. This is generally used to compress formats like Avro and sequence files.

When comes to an option to select a technique that is more beneficial to our use case when there is a large size file we should not use the technique in which the file is not splittable.

Conclusion

We are using Hadoop map-reduce algorithms and as we understood different compression techniques and the benefits of each of them we will be able to decide when to use these compression techniques and which compression technique best matches which scenario. Hope this will help you to make the right decision as and when such a scenario comes up front. Also, make a mental note that it is not necessary to

0

 

 

Comments

Post a Comment

Popular posts from this blog

what is polynomial regression

  Decision tree classification in R Decision trees are one of the most basic and widely used machine learning algorithms, which fall under supervised machine learning techniques. Decision trees can handle both regression and classification tasks, and therefore learning decision trees is a must for those who aspire to be data scientists. In this article, we will learn about decision trees, how to work with decision trees, and how to implement decision trees in R. We will also discuss the applications of decision trees along with their advantages and disadvantages What is a decision tree and how does it work? Decision trees are a non-parametric form of supervised machine learning algorithm used for both classification and regression. As the name suggests, decision trees work by asking a Boolean form of question and, based on the answer, make a decision that goes further in the form of a tree, thus the name decision tree. The model further asks the questions until the predict...
Read more

advantages and disadvantages of decision tree

  Various types of data compression in MapReduce When Hadoop word comes to mind instantly, one more word also comes side by side in mind which is big data. Big data means a very large amount of data. When we need to play with a large amount of data there will always be an issue of scarcity of space. So, how can we or Hadoop as architecture can handle such a critical issue? Hadoop has provided very nice and important to rescue us from this issue. The resolution is data compression. We can do data compression using different Hadoop libraries on our huge dataset. If you are still not clear about what are the benefits of data compression in Hadoop let me show you. As we will compress the dataset size required to store data will decrease drastically. On the other end as we all know we need to transfer data among the Hadoop clusters from one machine to another. So, as a result of data compression data size will decrease, and eventually, the speed at which data will be transferred over ...
Read more

decision tree in r

  https://www.dataspoof.info/post/decision-tree-classification-in-r/ Decision trees are one of the most basic and widely used machine learning algorithms, which fall under supervised machine learning techniques. Decision trees can handle both regression and classification tasks, and therefore learning decision trees is a must for those who aspire to be data scientists.
Read more