private static void addLeaves( Tap tap, Set<Tap> taps ) { if( tap instanceof CompositeTap ) { Iterator<? extends Tap> childTaps = ( (CompositeTap<? extends Tap>) tap ).getChildTaps(); while( childTaps.hasNext() ) addLeaves( childTaps.next(), taps ); } else { taps.add( tap ); } } }
private void unwrapCompositeTaps( LinkedList<Tap> taps ) { ListIterator<Tap> iterator = taps.listIterator(); while( iterator.hasNext() ) { Tap tap = iterator.next(); if( tap instanceof CompositeTap ) { iterator.remove(); Iterator<Tap> childTaps = ( (CompositeTap) tap ).getChildTaps(); while( childTaps.hasNext() ) { iterator.add( childTaps.next() ); iterator.previous(); // force cursor backward } } } } }
/** Determines the type of the configuration type argument of the supplied {@link Tap}. */ static Class<?> getTapConfigClass(Tap<?, ?, ?> tap) { Class<?> currentClass = tap.getClass(); if (CompositeTap.class.isAssignableFrom(currentClass)) { currentClass = ((CompositeTap<?>) tap).getChildTaps().next().getClass(); } while (currentClass != null) { if (Tap.class.isAssignableFrom(currentClass)) { Type genericSuperclass = currentClass.getGenericSuperclass(); if (genericSuperclass instanceof ParameterizedType) { ParameterizedType tapType = (ParameterizedType) genericSuperclass; Type[] typeParameters = tapType.getActualTypeArguments(); Type configTypeParameter = typeParameters[0]; if (configTypeParameter instanceof Class) { Class<?> configClassParameter = (Class<?>) configTypeParameter; return configClassParameter; } } } currentClass = currentClass.getSuperclass(); } return null; }
private MessageType readSchema(FlowProcess<JobConf> flowProcess, Tap tap) { try { Hfs hfs; if( tap instanceof CompositeTap ) hfs = (Hfs) ( (CompositeTap) tap ).getChildTaps().next(); else hfs = (Hfs) tap; List<Footer> footers = getFooters(flowProcess, hfs); if(footers.isEmpty()) { throw new TapException("Could not read Parquet metadata at " + hfs.getPath()); } else { return footers.get(0).getParquetMetadata().getFileMetaData().getSchema(); } } catch (IOException e) { throw new TapException(e); } }
private MessageType readSchema(FlowProcess<? extends JobConf> flowProcess, Tap tap) { try { Hfs hfs; if( tap instanceof CompositeTap ) hfs = (Hfs) ( (CompositeTap) tap ).getChildTaps().next(); else hfs = (Hfs) tap; List<Footer> footers = getFooters(flowProcess, hfs); if(footers.isEmpty()) { throw new TapException("Could not read Parquet metadata at " + hfs.getPath()); } else { return footers.get(0).getParquetMetadata().getFileMetaData().getSchema(); } } catch (IOException e) { throw new TapException(e); } }
private MessageType readSchema(FlowProcess<? extends JobConf> flowProcess, Tap tap) { try { Hfs hfs; if( tap instanceof CompositeTap ) hfs = (Hfs) ( (CompositeTap) tap ).getChildTaps().next(); else hfs = (Hfs) tap; List<Footer> footers = getFooters(flowProcess, hfs); if(footers.isEmpty()) { throw new TapException("Could not read Parquet metadata at " + hfs.getPath()); } else { return footers.get(0).getParquetMetadata().getFileMetaData().getSchema(); } } catch (IOException e) { throw new TapException(e); } }
private MessageType readSchema(FlowProcess<JobConf> flowProcess, Tap tap) { try { Hfs hfs; if( tap instanceof CompositeTap ) hfs = (Hfs) ( (CompositeTap) tap ).getChildTaps().next(); else hfs = (Hfs) tap; List<Footer> footers = getFooters(flowProcess, hfs); if(footers.isEmpty()) { throw new TapException("Could not read Parquet metadata at " + hfs.getPath()); } else { return footers.get(0).getParquetMetadata().getFileMetaData().getSchema(); } } catch (IOException e) { throw new TapException(e); } }
@Override public Fields retrieveSourceFields( FlowProcess<? extends Configuration> flowProcess, Tap tap ) { if( !skipHeader || !getSourceFields().isUnknown() ) return getSourceFields(); // no need to open them all if( tap instanceof CompositeTap ) tap = (Tap) ( (CompositeTap) tap ).getChildTaps().next(); // should revert to file:// (Lfs) if tap is Lfs if( tap instanceof TapWith ) tap = ( (TapWith) tap ).withScheme( new TextLine( new Fields( "line" ), charsetName ) ).asTap(); else tap = new Hfs( new TextLine( new Fields( "line" ), charsetName ), tap.getFullIdentifier( flowProcess ) ); setSourceFields( delimitedParser.parseFirstLine( flowProcess, tap ) ); return getSourceFields(); }
@Override public Fields retrieveSourceFields( FlowProcess<? extends Properties> process, Tap tap ) { if( !skipHeader || !getSourceFields().isUnknown() ) return getSourceFields(); // no need to open them all if( tap instanceof CompositeTap ) tap = (Tap) ( (CompositeTap) tap ).getChildTaps().next(); // can read either a single file, or search the directory for a file if( tap instanceof TapWith ) tap = ( (TapWith) tap ).withScheme( new TextLine( new Fields( "line" ), compressor, charsetName ) ).asTap(); else tap = new DirTap( new TextLine( new Fields( "line" ), compressor, charsetName ), tap.getIdentifier() ); setSourceFields( delimitedParser.parseFirstLine( process, tap ) ); return getSourceFields(); }
@Override public Fields retrieveSourceFields( FlowProcess<? extends Configuration> flowProcess, Tap tap ) { if( !skipHeader || !getSourceFields().isUnknown() ) return getSourceFields(); // no need to open them all if( tap instanceof CompositeTap ) tap = (Tap) ( (CompositeTap) tap ).getChildTaps().next(); // should revert to file:// (Lfs) if tap is Lfs if( tap instanceof TapWith ) tap = ( (TapWith) tap ).withScheme( new TextLine( new Fields( "line" ), charsetName ) ).asTap(); else tap = new Hfs( new TextLine( new Fields( "line" ), charsetName ), tap.getFullIdentifier( flowProcess ) ); setSourceFields( delimitedParser.parseFirstLine( flowProcess, tap ) ); return getSourceFields(); }