shellState.getAccumuloClient().tableOperations().importDirectory(dir) .to(shellState.getTableName()).tableTime(setTime).load(); } else if (args.length == 3) { String failureDir = args[1]; setTime = Boolean.parseBoolean(cl.getArgs()[2]); shellState.getAccumuloClient().tableOperations().importDirectory(shellState.getTableName(), dir, failureDir, setTime); return 0;
@Override public Object call() throws Exception { c.tableOperations().importDirectory(tableName, dir, err, false); return null; } }));
static public void bulkImport(Connector c, FileSystem fs, String table, String dir) throws Exception { String failDir = dir + "_failures"; Path failPath = new Path(failDir); fs.delete(failPath, true); fs.mkdirs(failPath); // Ensure server can read/modify files c.tableOperations().importDirectory(table, dir, failDir, false); if (fs.listStatus(failPath).length > 0) { throw new Exception("Some files failed to bulk import"); } }
@Override public Object call() throws Exception { c.tableOperations().importDirectory(tableName, dir, err, false); return null; } }));
private void bulkImport(FileSystem fs, State state, Environment env, String tableName, String rootDir, String prefix) throws Exception { while (true) { String bulkDir = rootDir + "/" + prefix + "_bulk"; String failDir = rootDir + "/" + prefix + "_failure"; Path failPath = new Path(failDir); fs.delete(failPath, true); fs.mkdirs(failPath); env.getConnector().tableOperations().importDirectory(tableName, bulkDir, failDir, true); FileStatus[] failures = fs.listStatus(failPath); if (failures != null && failures.length > 0) { log.warn("Failed to bulk import some files, retrying "); for (FileStatus failure : failures) { if (!failure.getPath().getName().endsWith(".seq")) fs.rename(failure.getPath(), new Path(new Path(bulkDir), failure.getPath().getName())); else log.debug("Ignoring " + failure.getPath()); } sleepUninterruptibly(3, TimeUnit.SECONDS); } else break; } }
@Override public int execute(final String fullCommand, final CommandLine cl, final Shell shellState) throws IOException, AccumuloException, AccumuloSecurityException, TableNotFoundException { shellState.checkTableState(); String dir = cl.getArgs()[0]; String failureDir = cl.getArgs()[1]; final boolean setTime = Boolean.parseBoolean(cl.getArgs()[2]); shellState.getConnector().tableOperations().importDirectory(shellState.getTableName(), dir, failureDir, setTime); return 0; }
tops.importDirectory(tableName, tableDir.toString(), failuresDir, false); log.info("Completed bringing map files online for " + tableName); validateComplete();
private int loadBulkFiles() throws IOException, AccumuloException, AccumuloSecurityException, TableNotFoundException { Configuration conf = getConf(); Connector connector = WikipediaConfiguration.getConnector(conf); FileSystem fs = FileSystem.get(conf); String directory = WikipediaConfiguration.bulkIngestDir(conf); String failureDirectory = WikipediaConfiguration.bulkIngestFailureDir(conf); for(FileStatus status: fs.listStatus(new Path(directory))) { if(status.isDir() == false) continue; Path dir = status.getPath(); Path failPath = new Path(failureDirectory+"/"+dir.getName()); fs.mkdirs(failPath); connector.tableOperations().importDirectory(dir.getName(), dir.toString(), failPath.toString(), true); } return 0; }
@Test public void testBulkImportFailure() throws Exception { String tableName = getUniqueNames(1)[0]; TableOperations to = getConnector().tableOperations(); to.create(tableName); FileSystem fs = getFileSystem(); Path rootPath = new Path(cluster.getTemporaryPath(), getClass().getName()); Path bulk = new Path(rootPath, "bulk"); log.info("bulk: {}", bulk); if (fs.exists(bulk)) { fs.delete(bulk, true); } assertTrue(fs.mkdirs(bulk)); Path err = new Path(rootPath, "err"); log.info("err: {}", err); if (fs.exists(err)) { fs.delete(err, true); } assertTrue(fs.mkdirs(err)); Path bogus = new Path(bulk, "bogus.rf"); fs.create(bogus).close(); log.info("bogus: {}", bogus); assertTrue(fs.exists(bogus)); log.info("Importing {} into {} with failures directory {}", bulk, tableName, err); to.importDirectory(tableName, bulk.toString(), err.toString(), false); assertEquals(1, fs.listStatus(err).length); }
@Override public void importDirectory(ByteBuffer login, String tableName, String importDir, String failureDir, boolean setTime) throws org.apache.accumulo.proxy.thrift.TableNotFoundException, org.apache.accumulo.proxy.thrift.AccumuloException, org.apache.accumulo.proxy.thrift.AccumuloSecurityException, TException { try { getConnector(login).tableOperations().importDirectory(tableName, importDir, failureDir, setTime); } catch (Exception e) { handleExceptionTNF(e); } }
fs.delete(failures, true); fs.mkdirs(new Path(workDir, "failures")); connector.tableOperations().importDirectory(tableName, workDir + "/files", workDir + "/failures", false);
public static void main(String[] args) throws IOException, AccumuloException, AccumuloSecurityException, TableNotFoundException { final FileSystem fs = FileSystem.get(CachedConfiguration.getInstance()); Opts opts = new Opts(); if (args.length == 5) { System.err.println( "Deprecated syntax for BulkImportDirectory, please use the new style (see --help)"); final String user = args[0]; final byte[] pass = args[1].getBytes(UTF_8); final String tableName = args[2]; final String dir = args[3]; final String failureDir = args[4]; final Path failureDirPath = new Path(failureDir); fs.delete(failureDirPath, true); fs.mkdirs(failureDirPath); HdfsZooInstance.getInstance().getConnector(user, new PasswordToken(pass)).tableOperations() .importDirectory(tableName, dir, failureDir, false); } else { opts.parseArgs(BulkImportDirectory.class.getName(), args); fs.delete(new Path(opts.failures), true); fs.mkdirs(new Path(opts.failures)); opts.getConnector().tableOperations().importDirectory(opts.getTableName(), opts.source, opts.failures, false); } } }
@Override public int run(final String[] strings) throws Exception { LOGGER.info("Ensuring table {} exists", store.getTableName()); TableUtils.ensureTableExists(store); // Hadoop configuration final Configuration conf = getConf(); final FileSystem fs = FileSystem.get(conf); checkHdfsDirectories(failurePath, fs); // Remove the _SUCCESS file to prevent warning in Accumulo LOGGER.info("Removing file {}/_SUCCESS", inputPath); fs.delete(new Path(inputPath + "/_SUCCESS"), false); // Set all permissions if (options == null || !Boolean.parseBoolean(options.get(AccumuloProperties.HDFS_SKIP_PERMISSIONS))) { IngestUtils.setDirectoryPermsForAccumulo(fs, new Path(inputPath)); } // Import the files LOGGER.info("Importing files in {} to table {}", inputPath, store.getTableName()); store.getConnection().tableOperations().importDirectory(store.getTableName(), inputPath, failurePath, false); return SUCCESS_RESPONSE; }
conn.tableOperations().importDirectory(tableName, bulkDir, bulkDir + "_f", rand.nextBoolean());
env.getConnector().tableOperations().importDirectory(Setup.getTableName(), dir.toString(), fail.toString(), true); fs.delete(dir, true);
to.importDirectory(tableName, bulk.toString(), err.toString(), false);
childTableOperations.importDirectory(normalizedChildTableName, files.toString(), failures.toString(), false);
c.tableOperations().importDirectory(tableName, files.toString(), bulkFailures.toString(), false); VerifyIngest.Opts vopts = new VerifyIngest.Opts();
/** * Imports the child files that hold the table data into the parent instance as a temporary table. * @param childTableName the name of the child table to import into a temporary parent table. * @throws Exception */ public void importChildFilesToTempParentTable(final String childTableName) throws Exception { // Create a temporary table in the parent instance to import the child files to. Then run the merge process on the parent table and temp child table. final String tempChildTable = childTableName + TEMP_SUFFIX; createTempTableIfNeeded(tempChildTable); final AccumuloRdfConfiguration parentAccumuloRdfConfiguration = new AccumuloRdfConfiguration(conf); parentAccumuloRdfConfiguration.setTablePrefix(childTablePrefix); final Connector parentConnector = AccumuloRyaUtils.setupConnector(parentAccumuloRdfConfiguration); final TableOperations parentTableOperations = parentConnector.tableOperations(); final Path localWorkDir = CopyTool.getPath(localMergeFileImportDir, childTableName); final Path hdfsBaseWorkDir = CopyTool.getPath(baseImportDir, childTableName); CopyTool.copyLocalToHdfs(localWorkDir, hdfsBaseWorkDir, conf); final Path files = CopyTool.getPath(hdfsBaseWorkDir.toString(), "files"); final Path failures = CopyTool.getPath(hdfsBaseWorkDir.toString(), "failures"); final FileSystem fs = FileSystem.get(conf); // With HDFS permissions on, we need to make sure the Accumulo user can read/move the files fs.setPermission(hdfsBaseWorkDir, new FsPermission(FsAction.ALL, FsAction.ALL, FsAction.ALL)); if (fs.exists(failures)) { fs.delete(failures, true); } fs.mkdirs(failures); parentTableOperations.importDirectory(tempChildTable, files.toString(), failures.toString(), false); AccumuloRyaUtils.printTablePretty(tempChildTable, conf); }
c.tableOperations().importDirectory(tableName, files.toString(), bulkFailures.toString(), true); double diffSeconds = (System.currentTimeMillis() - now) / 1000.; log.info(String.format("Import took %.2f seconds", diffSeconds));