flume - Using Kafka to import data to Hadoop -

-

At first I was wondering if incidents should be used to obtain in Hadop, where they will be stored and From time to time analysis will be done on them (possibly using Ooozie to periodically analyze) Kafka or flu, and decided that Kafka is probably a better solution, because we have a component that processes the event So, in this way, both batch and Event processing components get data in the same

but know that I'm curtly looking for suggestions on how to get data from Broker to Hadoop.

I found that Flume can be used in conjunction with kafka

  • Fluid - Includes the source (consumer) and sync (manufacturer)

And it has also been found on the same page and has a name in it

  • Camus - LinkedIn's Kaafka => HDFS pipeline This is a link for all data usage Is done, and works great.

I'm interested in what would be better (and easier, better document solution) to do this? Also, are there any examples or tutorials how to do this?

When I want to simplify this type of use, high-level consumer ?

I have been opened for suggestions if there is another / better solution than this two.

Thank you

You use the flu to dump the data from Kfca to HDFS can do. The following is an example of the fact that the source of the flu is the source and the sink is its property file change.

Step:

  1. Create a cool theme Cafa theme - Creating - Zucker localhost: 2181 cuff console creator Kafka-console-producer Use the above mentioned subject to write on.

    3. < / Li>

    flume1.sources = Kafka-source-1
    flume1.channels = hdfs-channel-1
    flume1.sinks = hdfs-sink-1
    flume1.sources. Kafka-source-1.type = Org.apache.flume.source.kafka.KafkaSource
    flume1.sources.kafka-source-1.zookeeperConnect = Localhost: 2181
    flume1.sources.kafka-source-1. Topic = testkafka
    flume1 .sources.kafka-source-1.batchSize = 100
    flume1.sources.kafka-source-1.channels = hdfs-channel-1

    flume1 .channels.hdfs-channel- 1. Type = Memory-flume1.sinks.hdfs-sink-1.channel = hdfs-channel-1
    Flu 1.sinks.hdfs-sink-1.type = hdfs
    Flume1.sinks.hdfs-sink-1.hdfs.writeFormat = text
    flume1.sinks.hdfs-sink-1.hdfs.fileType = datastream flume1.sinks.hdfs-sink-1.hdfs.filePrefix = Test-events
    flume1.sinks.hdfs-sink-1.hdfs.useLocalTimeStamp = true
    flu Me1.sinks.hdfs-sink-1 Hdfs.path = / tmp / cuff /% {subject} /% y-% m-% d
    flume1.sinks.hdfs-sink-1.hdfs.rollCount = 100 < Br> flume1.sinks.hdfs-sync1.hdfs.rollSize = 0
    flume1.channels.hdfs-channel-1.capacity = 10000
    flume1.channels.hdfs-channel-1.transactionCapacity = 1000 < Save the above config file as example.conf

    1. Flom agent flume-ng agent -n flume1 -c conf -f example.conf - run Dflume.root

      / tmp / kafka /% {subject} / /

      2. < / Li>% Y-% m-% d


Comments

Post a Comment

Popular posts from this blog

apache - 504 Gateway Time-out The server didn't respond in time. How to fix it? -

-
Using a form submission on an embedded iframe , the customer downloads a compressed log file Requested. The request was sent to the server, which is the compressed log files, perform some database operations and returns a compressed file. After just 2 minutes, 504 gateway time-out server did not respond timely message on browser net panel How to fix this error? The log files were taking a long time to compress, and timeout was set to 2min The error was fixed by extending the file file: # # timeout: The number of seconds before getting the time out. # # # Timeout 120 timeout 600
Read more

c# - .net WebSocket: CloseOutputAsync vs CloseAsync -

-
We have a working ASP.NET Web API REST service, which is one of the methods of our controller, HTTPTTEX. .) Socket handler code looks something like this ... Public async task socket handler (AspNetWebSocketContext context) {_webSocket = context.WebSocket; ... while (! Cts.IsCancellationRequested) {WebSocketReceiveResult Results = _webSocket.ReceiveAsync (Input Segment, cts.Token) .Result; WebSocketsStateCollusocketState = _webSocket.State; If (result.MessageType == WebSocketMessageType.Close || currentSocketState == WebSocketState.CloseReceived) {// What should I use. CloseAysnc () or. CloseOutputAsync ()? _webSocket.CloseOutputAsync (WebSocketCloseStatus.NormalClosure, "Client Requested", cts.Token). Wait (); } If (currentSocketState == WebSocketState.Open) {...}}} .What is the difference between .CooseAsync () and CloseOutputAysnc ()? I tried both of them and they both seemed to work fine but some difference should be the same they both describe very similar to...
Read more

c++ - How to properly scale qgroupbox title with stylesheet for high resolution display? -

-
I am trying to apply a stylesheet for QGroupBox (QT4.8), which works on the normal screen ( 96 dpi) high resolution screen (Yoga 2 Pro, 3200x1800, 275 dpi, windows 8.1). The following style looks good on the 275 dpi screen, but the top margins on a regular 96 dpi screen are far too big. QGroupBox {border: 1px solid red; Range radius: 7px; Margin-top: 12x; } QGroupBox :: Title {subcontrol-origin: margin; Subcontrol-position: left above; Padding-left: 10px; Padding-right: 10px; } Changing the top-margin has an effect, but I did not get a proper setting which works on both screens. If I shorten the value, the content of the group box overlaps with the title on 275 dpi screen. I was also playing with units "East", "PX", "MX", "PT". Reading the document I would have guessed, "2 X" was the correct solution, which should be scaled with different screen resolutions. Without the stylesheet, the groupbox works well on both screens. ...
Read more