protected HiveDatasetFinder(FileSystem fs, Properties properties, HiveMetastoreClientPool clientPool, EventSubmitter eventSubmitter) throws IOException { this(fs, properties, clientPool, eventSubmitter, ConfigClientCache.getClient(VersionStabilityPolicy.STRONG_LOCAL_STABILITY)); }
/** * Create the {@link ConfigClient} based on the {@link VersionStabilityPolicy}. * @param policy - {@link VersionStabilityPolicy} to specify the stability policy which control the caching layer creation * @return - {@link ConfigClient} for client to use to access the {@link ConfigStore} */ public static ConfigClient createConfigClient(VersionStabilityPolicy policy) { return new ConfigClient(policy); }
@Override public ConfigClient call() throws Exception { return ConfigClient.createConfigClient(policy); } });
public Config getConfig(String configKeyStr, Optional<Config> runtimeConfig) throws ConfigStoreFactoryDoesNotExistsException, ConfigStoreCreationException, VersionDoesNotExistException, URISyntaxException { return this.getConfig(new URI(configKeyStr), runtimeConfig); }
public Collection<URI> getImports(URI configKeyUri, boolean recursive, Optional<Config> runtimeConfig) throws ConfigStoreFactoryDoesNotExistsException, ConfigStoreCreationException, VersionDoesNotExistException { ConfigStoreAccessor accessor = this.getConfigStoreAccessor(configKeyUri); ConfigKeyPath configKeypath = ConfigClientUtils.buildConfigKeyPath(configKeyUri, accessor.configStore); Collection<ConfigKeyPath> result; if (!recursive) { result = accessor.topologyInspector.getOwnImports(configKeypath, runtimeConfig); } else { result = accessor.topologyInspector.getImportsRecursively(configKeypath, runtimeConfig); } return ConfigClientUtils.buildUriInClientFormat(result, accessor.configStore, configKeyUri.getAuthority() != null); }
/** * Get the URIs which imports the input URI * * @param configKeyUri - The URI for the configuration key. * @param recursive - Specify whether to get direct or recursively imported by links * @return the URIs which imports the input URI * * @throws ConfigStoreFactoryDoesNotExistsException: if missing scheme name or the scheme name is invalid * @throws ConfigStoreCreationException: Specified {@link ConfigStoreFactory} can not create required {@link ConfigStore} * @throws VersionDoesNotExistException: Required version does not exist anymore ( may get deleted by retention job ) */ public Collection<URI> getImportedBy(URI configKeyUri, boolean recursive) throws ConfigStoreFactoryDoesNotExistsException, ConfigStoreCreationException, VersionDoesNotExistException { return getImportedBy(configKeyUri, recursive, Optional.<Config>absent()); }
public Config getConfig(URI configKeyUri, Optional<Config> runtimeConfig) throws ConfigStoreFactoryDoesNotExistsException, ConfigStoreCreationException, VersionDoesNotExistException { ConfigStoreAccessor accessor = this.getConfigStoreAccessor(configKeyUri); ConfigKeyPath configKeypath = ConfigClientUtils.buildConfigKeyPath(configKeyUri, accessor.configStore); return accessor.valueInspector.getResolvedConfig(configKeypath, runtimeConfig); }
/** * * @param configKeyURI - URI provided by client , which could missing authority/store root directory * @param cs - ConfigStore corresponding to the input URI. Require input URI's scheme/authority name * match ConfigStore's scheme/authority * @return - {@link ConfigKeyPath} for the relative path */ public static ConfigKeyPath buildConfigKeyPath(URI configKeyURI, ConfigStore cs) { checkMatchingSchemeAndAuthority(configKeyURI, cs); // Example store root is etl-hdfs://eat1-nertznn01.grid.linkedin.com:9000/user/mitu/HdfsBasedConfigTest URI relative = cs.getStoreURI().relativize(configKeyURI); return getConfigKeyPath(relative.getPath()); }
@Test (expectedExceptions = java.lang.IllegalArgumentException.class ) public void testInvalidSchemeURI() throws URISyntaxException { URI clientURI = new URI("hdfs:///datasets/a1/a2"); ConfigClientUtils.buildConfigKeyPath(clientURI, mockConfigStore); }
private URI getMatchedFloorKeyFromCache(URI configKeyURI) { URI floorKey = this.configStoreAccessorMap.floorKey(configKeyURI); if (floorKey == null) { return null; } // both scheme name and authority name, if present, should match // or both authority should be null if (ConfigClientUtils.isAncestorOrSame(configKeyURI, floorKey)) { return floorKey; } return null; }
private ConfigClient(VersionStabilityPolicy policy) { this(policy, new ConfigStoreFactoryRegister()); }
/** * Get the import links of the input URI. * * @param configKeyUri - The URI for the configuration key. * @param recursive - Specify whether to get direct import links or recursively import links * @return the import links of the input URI. * * @throws ConfigStoreFactoryDoesNotExistsException: if missing scheme name or the scheme name is invalid * @throws ConfigStoreCreationException: Specified {@link ConfigStoreFactory} can not create required {@link ConfigStore} * @throws VersionDoesNotExistException: Required version does not exist anymore ( may get deleted by retention job ) */ public Collection<URI> getImports(URI configKeyUri, boolean recursive) throws ConfigStoreFactoryDoesNotExistsException, ConfigStoreCreationException, VersionDoesNotExistException { return getImports(configKeyUri, recursive, Optional.<Config>absent()); }
public static List<ConfigKeyPath> getConfigKeyPath(List<String> input) { List<ConfigKeyPath> result = new ArrayList<>(); for (String s : input) { result.add(getConfigKeyPath(s)); } return result; }
/** * Convenient method to get resolved {@link Config} based on String input. */ public Config getConfig(String configKeyStr) throws ConfigStoreFactoryDoesNotExistsException, ConfigStoreCreationException, VersionDoesNotExistException, URISyntaxException { return getConfig(configKeyStr, Optional.<Config>absent()); }
private synchronized static ConfigClient getConfigClient() { if (CONFIG_CLIENT == null || !CONFIG_CLIENT.isValid()) { CONFIG_CLIENT = new TTLResourceEntry<>(ConfigClient.createConfigClient(VersionStabilityPolicy.READ_FRESHEST), CONFIG_CLIENT_TTL_IN_MILLIS, false); } return CONFIG_CLIENT.getResource(); } }
public Collection<URI> getImportedBy(URI configKeyUri, boolean recursive, Optional<Config> runtimeConfig) throws ConfigStoreFactoryDoesNotExistsException, ConfigStoreCreationException, VersionDoesNotExistException { ConfigStoreAccessor accessor = this.getConfigStoreAccessor(configKeyUri); ConfigKeyPath configKeypath = ConfigClientUtils.buildConfigKeyPath(configKeyUri, accessor.configStore); Collection<ConfigKeyPath> result; if (!recursive) { result = accessor.topologyInspector.getImportedBy(configKeypath, runtimeConfig); } else { result = accessor.topologyInspector.getImportedByRecursively(configKeypath, runtimeConfig); } return ConfigClientUtils.buildUriInClientFormat(result, accessor.configStore, configKeyUri.getAuthority() != null); }
public ManagedCleanableDatasetFinder(FileSystem fs, Properties jobProps, Config config) { this(fs, jobProps, config, ConfigClientCache.getClient(VersionStabilityPolicy.STRONG_LOCAL_STABILITY)); }
@Test (expectedExceptions = java.lang.IllegalArgumentException.class ) public void testInvalidAuthortiyURI() throws URISyntaxException { URI clientURI = new URI("etl-hdfs://ltx1-nertznn01.grid.linkedin.com:9000/user/mitu/HdfsBasedConfigTest/datasets/a1/a2"); ConfigClientUtils.buildConfigKeyPath(clientURI, mockConfigStore); } }
/** * Wrapper to convert Checked Exception to Unchecked Exception * Easy to use in lambda expressions */ public static Config getConfig(ConfigClient client, URI u, Optional<Config> runtimeConfig) { try { return client.getConfig(u, runtimeConfig); } catch (ConfigStoreFactoryDoesNotExistsException | ConfigStoreCreationException e) { throw new Error(e); } }
@Override public Void call() throws Exception { // Process each {@link Config}, find dataset and add those into the datasets Config c = confClient.getConfig(u); List<Dataset> datasetForConfig = new ConfigBasedMultiDatasets(c, p, blacklistPatterns).getConfigBasedDatasetList(); datasets.addAll(datasetForConfig); return null; } };