7 things about Datalake you must know before getting started in 2023

Datalake is the central place to store big data from multiple sources.

Datalake can be used to store structured, semi-structured, or unstructured data.

[{"selector":"#anim-c019b35d-c122-4eff-bc7a-579bf9dcb8bf","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-2bed9729-d35b-4858-a063-e9a8ced21f25","keyframes":{"transform":["translate3d(0px, 182.20488%, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}]

Datalake is usually deployed on commodity hardware and can scale easily as per the user's need.

[{"selector":"#anim-612a557b-c662-442c-a437-6ece51e78366","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-b95dede1-0d87-470d-86f7-5dc53679711f","keyframes":{"transform":["translate3d(0px, 149.20635%, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}]

Datalake can be the setup of cloud and on-premises as per the requirement.

[{"selector":"#anim-3ec2aa52-a5b4-4b38-8b04-16d77f233abf","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-c4570e4c-2ae0-409a-a680-b984677e36da","keyframes":{"transform":["translate3d(0px, 221.06485%, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}]

Managing datalake on on-premises requires the team to integrate lots of complex components but it gives users the flexibility to customise as per their needs.

[{"selector":"#anim-a1da2f3a-4308-41d6-bb4b-dc96eacb1a5e","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-3d0205b5-3a17-4e76-95ed-c7d4898d94fb","keyframes":{"transform":["translate3d(0px, 125.76204%, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}]

Cloud data lake is a great solution if you do need want to set up your infrastructure. However, it is very costly as you need to pay for storage, bandwidth, etc.

[{"selector":"#anim-baaf49b5-13ce-4353-9249-7dd5acfed9e8","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-addc9b68-c0b5-4af6-a0e2-a6c8138ace4e","keyframes":{"transform":["translate3d(0px, 135.05408%, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}]

The key components of datalake are Hadoop, spark,Hbase, MongoDB, Minio, druid,etc

[{"selector":"#anim-0388c066-ab40-4fbc-b5d8-7f14c840844d","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-793f4000-60d4-4adc-badb-e79119b4c76a","keyframes":{"transform":["translate3d(0px, 225.95489%, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}]

7 things you should know about apache kafka

[{"selector":"#anim-b71417b0-fba2-40ce-99a9-0987f6957379","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-c5850727-b186-4189-bbde-f8c498a23687","keyframes":{"transform":["translate3d(0px, 303.09826%, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}]