How to use
setHdfsUri
method
in
org.apache.storm.hdfs.spout.HdfsSpout

Best Java code snippets using org.apache.storm.hdfs.spout.HdfsSpout.setHdfsUri (Showing top 3 results out of 315)

Add the Codota plugin to your IDE and get smart completions

private void myMethod () {
 SimpleDateFormat s =String pattern;new SimpleDateFormat(pattern)
String template;Locale locale;new SimpleDateFormat(template, locale)
new SimpleDateFormat()
Smart code suggestions by Codota
}

.setHdfsUri(hdfsUri)
.setSourceDir(sourceDir)
.setArchiveDir(archiveDir)

private AutoCloseableHdfsSpout makeSpout(String readerType, String[] outputFields) {
  HdfsSpout spout = new HdfsSpout().withOutputFields(outputFields)
                   .setReaderType(readerType)
                   .setHdfsUri(DFS_CLUSTER_RULE.getDfscluster().getURI().toString())
                   .setSourceDir(source.toString())
                   .setArchiveDir(archive.toString())
                   .setBadFilesDir(badfiles.toString());
  return new AutoCloseableHdfsSpout(spout);
}

static StormTopology getTopology(Map<String, Object> config) {
  final int spoutNum = Helper.getInt(config, SPOUT_NUM, DEFAULT_SPOUT_NUM);
  final int boltNum = Helper.getInt(config, BOLT_NUM, DEFAULT_BOLT_NUM);
  final String fileFormat = Helper.getStr(config, "text");
  final String hdfsUri = Helper.getStr(config, HDFS_URI);
  final String sourceDir = Helper.getStr(config, SOURCE_DIR);
  final String archiveDir = Helper.getStr(config, ARCHIVE_DIR);
  final String badDir = Helper.getStr(config, BAD_DIR);
  // 1 -  Setup Hdfs Spout   --------
  HdfsSpout spout = new HdfsSpout()
    .setReaderType(fileFormat)
    .setHdfsUri(hdfsUri)
    .setSourceDir(sourceDir)
    .setArchiveDir(archiveDir)
    .setBadFilesDir(badDir)
    .withOutputFields(TextFileReader.defaultFields);
  // 2 -   DevNull Bolt   --------
  DevNullBolt bolt = new DevNullBolt();
  // 3 - Setup Topology  --------
  TopologyBuilder builder = new TopologyBuilder();
  builder.setSpout(SPOUT_ID, spout, spoutNum);
  builder.setBolt(BOLT_ID, bolt, boltNum)
      .localOrShuffleGrouping(SPOUT_ID);
  return builder.createTopology();
}

Popular methods of HdfsSpout

<init>
setArchiveDir
setBadFilesDir
setReaderType
setSourceDir
withOutputFields
Output field names. Number of fields depends upon the reader type
ack
canCommitNow
checkValidReader
close
closeReaderAndResetTrackers
commitProgress

Popular in Java

Parsing JSON documents to java classes using gson
getApplicationContext (Context)
getOriginalFilename (MultipartFile)
Return the original filename in the client's filesystem.This may contain path information depending
scheduleAtFixedRate (ScheduledExecutorService)
Creates and executes a periodic action that becomes enabled first after the given initial delay, and
ByteBuffer (java.nio)
A buffer for bytes. A byte buffer can be created in either one of the following ways: * #allocate(i
MessageDigest (java.security)
Uses a one-way hash function to turn an arbitrary number of bytes into a fixed-length byte sequence.
MessageFormat (java.text)
MessageFormat provides a means to produce concatenated messages in language-neutral way. Use this to
List (java.util)
A List is a collection which maintains an ordering for its elements. Every element in the List has a
JOptionPane (javax.swing)
Logger (org.slf4j)
The main user interface to logging. It is expected that logging takes place through concrete impleme

How to use setHdfsUrimethodin org.apache.storm.hdfs.spout.HdfsSpout

Best Java code snippets using org.apache.storm.hdfs.spout.HdfsSpout.setHdfsUri (Showing top 3 results out of 315)

How to use
setHdfsUri
method
in
org.apache.storm.hdfs.spout.HdfsSpout