public static void appendPrimaryIndexTypes(Dataset dataset, IAType itemType, IAType metaItemType, List<Object> target) throws AlgebricksException { ARecordType recordType = (ARecordType) itemType; ARecordType metaRecordType = (ARecordType) metaItemType; target.addAll(KeyFieldTypeUtil.getPartitoningKeyTypes(dataset, recordType, metaRecordType)); // Adds data record type. target.add(itemType); // Adds meta record type if any. if (dataset.hasMetaPart()) { target.add(metaItemType); } }
private static int[] getPrimaryKeyPermutationForUpsert(Dataset dataset) { // upsertIndicatorVar + prev record int f = 2; // add the previous meta second if (dataset.hasMetaPart()) { f++; } // add the previous filter third int numFilterFields = DatasetUtil.getFilterField(dataset) == null ? 0 : 1; if (numFilterFields > 0) { f++; } int numPrimaryKeys = dataset.getPrimaryKeys().size(); int[] pkIndexes = new int[numPrimaryKeys]; for (int i = 0; i < pkIndexes.length; i++) { pkIndexes[i] = f; f++; } return pkIndexes; }
private static IOperatorDescriptor createPrimaryIndexUpsertOp(JobSpecification spec, MetadataProvider metadataProvider, Dataset source, Dataset target) throws AlgebricksException { int numKeys = source.getPrimaryKeys().size(); int numValues = source.hasMetaPart() ? 2 : 1; int[] fieldPermutation = IntStream.range(0, numKeys + numValues).toArray(); Pair<IOperatorDescriptor, AlgebricksPartitionConstraint> upsertOpAndConstraints = DatasetUtil.createPrimaryIndexUpsertOp(spec, metadataProvider, target, source.getPrimaryRecordDescriptor(metadataProvider), fieldPermutation, MissingWriterFactory.INSTANCE); IOperatorDescriptor upsertOp = upsertOpAndConstraints.first; AlgebricksPartitionConstraintHelper.setPartitionConstraintInJobSpec(spec, upsertOp, upsertOpAndConstraints.second); return upsertOp; }
public static int[] createBTreeFieldsWhenThereisAFilter(Dataset dataset) { if (dataset.getDatasetType() == DatasetType.EXTERNAL) { return null; } List<String> filterField = getFilterField(dataset); if (filterField == null) { return null; } List<List<String>> partitioningKeys = dataset.getPrimaryKeys(); int valueFields = dataset.hasMetaPart() ? 2 : 1; int[] btreeFields = new int[partitioningKeys.size() + valueFields]; for (int i = 0; i < btreeFields.length; ++i) { btreeFields[i] = i; } return btreeFields; }
public static ARecordType getMetaType(MetadataProvider metadataProvider, Dataset dataset) throws AlgebricksException { if (dataset.hasMetaPart()) { return (ARecordType) metadataProvider.findType(dataset.getMetaItemTypeDataverseName(), dataset.getMetaItemTypeName()); } return null; }
public ITypeTraits[] getPrimaryTypeTraits(MetadataProvider metadataProvider, ARecordType recordType, ARecordType metaType) throws AlgebricksException { IStorageComponentProvider storageComponentProvider = metadataProvider.getStorageComponentProvider(); ITypeTraitProvider ttProvider = storageComponentProvider.getTypeTraitProvider(); List<List<String>> partitioningKeys = getPrimaryKeys(); int numPrimaryKeys = partitioningKeys.size(); ITypeTraits[] typeTraits = new ITypeTraits[numPrimaryKeys + 1 + (hasMetaPart() ? 1 : 0)]; List<Integer> indicators = null; if (hasMetaPart()) { indicators = ((InternalDatasetDetails) getDatasetDetails()).getKeySourceIndicator(); } for (int i = 0; i < numPrimaryKeys; i++) { IAType keyType = datasetType == DatasetType.EXTERNAL ? IndexingConstants.getFieldType(i) : (indicators == null || indicators.get(i) == 0) ? recordType.getSubFieldType(partitioningKeys.get(i)) : metaType.getSubFieldType(partitioningKeys.get(i)); typeTraits[i] = ttProvider.getTypeTrait(keyType); } typeTraits[numPrimaryKeys] = ttProvider.getTypeTrait(recordType); if (hasMetaPart()) { typeTraits[numPrimaryKeys + 1] = ttProvider.getTypeTrait(metaType); } return typeTraits; }
private void setPrimaryRecDescAndComparators() throws AlgebricksException { List<List<String>> partitioningKeys = dataset.getPrimaryKeys(); ISerializerDeserializer[] primaryRecFields = new ISerializerDeserializer[numPrimaryKeys + 1 + (dataset.hasMetaPart() ? 1 : 0)]; ITypeTraits[] primaryTypeTraits = new ITypeTraits[numPrimaryKeys + 1 + (dataset.hasMetaPart() ? 1 : 0)]; primaryComparatorFactories = new IBinaryComparatorFactory[numPrimaryKeys]; primaryBloomFilterKeyFields = new int[numPrimaryKeys]; ISerializerDeserializerProvider serdeProvider = metadataProvider.getDataFormat().getSerdeProvider(); List<Integer> indicators = null; if (dataset.hasMetaPart()) { indicators = ((InternalDatasetDetails) dataset.getDatasetDetails()).getKeySourceIndicator(); } for (int i = 0; i < numPrimaryKeys; i++) { IAType keyType = (indicators == null || indicators.get(i) == 0) ? itemType.getSubFieldType(partitioningKeys.get(i)) : metaType.getSubFieldType(partitioningKeys.get(i)); primaryRecFields[i] = serdeProvider.getSerializerDeserializer(keyType); primaryComparatorFactories[i] = BinaryComparatorFactoryProvider.INSTANCE.getBinaryComparatorFactory(keyType, true); primaryTypeTraits[i] = TypeTraitProvider.INSTANCE.getTypeTrait(keyType); primaryBloomFilterKeyFields[i] = i; } primaryRecFields[numPrimaryKeys] = payloadSerde; primaryTypeTraits[numPrimaryKeys] = TypeTraitProvider.INSTANCE.getTypeTrait(itemType); if (dataset.hasMetaPart()) { primaryRecFields[numPrimaryKeys + 1] = payloadSerde; primaryTypeTraits[numPrimaryKeys + 1] = TypeTraitProvider.INSTANCE.getTypeTrait(metaType); } primaryRecDesc = new RecordDescriptor(primaryRecFields, primaryTypeTraits); }
protected AlgebricksMetaOperatorDescriptor createCastOp(JobSpecification spec, DatasetType dsType, boolean strictCast) throws AlgebricksException { int[] outColumns = new int[1]; int[] projectionList = new int[(dataset.hasMetaPart() ? 2 : 1) + numPrimaryKeys]; int recordIdx; //external datascan operator returns a record as the first field, instead of the last in internal case if (dsType == DatasetType.EXTERNAL) { recordIdx = 0; outColumns[0] = 0; } else { recordIdx = numPrimaryKeys; outColumns[0] = numPrimaryKeys; } for (int i = 0; i <= numPrimaryKeys; i++) { projectionList[i] = i; } if (dataset.hasMetaPart()) { projectionList[numPrimaryKeys + 1] = numPrimaryKeys + 1; } IScalarEvaluatorFactory[] castEvalFact = new IScalarEvaluatorFactory[] { new ColumnAccessEvalFactory(recordIdx) }; IScalarEvaluatorFactory[] sefs = new IScalarEvaluatorFactory[1]; sefs[0] = createCastFunction(strictCast).createEvaluatorFactory(castEvalFact); AssignRuntimeFactory castAssign = new AssignRuntimeFactory(outColumns, sefs, projectionList); castAssign.setSourceLocation(sourceLoc); return new AlgebricksMetaOperatorDescriptor(spec, 1, 1, new IPushRuntimeFactory[] { castAssign }, new RecordDescriptor[] { enforcedRecDesc }); }
/** * Gets the comparator factories for the primary key fields of this dataset. * * @param metadataProvider, * the metadata provider. * @return the comparator factories for the primary key fields of this dataset. * @throws AlgebricksException */ public IBinaryComparatorFactory[] getPrimaryComparatorFactories(MetadataProvider metadataProvider, ARecordType recordType, ARecordType metaType) throws AlgebricksException { IStorageComponentProvider storageComponentProvider = metadataProvider.getStorageComponentProvider(); IBinaryComparatorFactoryProvider cmpFactoryProvider = storageComponentProvider.getComparatorFactoryProvider(); List<List<String>> partitioningKeys = getPrimaryKeys(); int numPrimaryKeys = partitioningKeys.size(); IBinaryComparatorFactory[] cmpFactories = new IBinaryComparatorFactory[numPrimaryKeys]; List<Integer> indicators = null; if (hasMetaPart()) { indicators = ((InternalDatasetDetails) getDatasetDetails()).getKeySourceIndicator(); } for (int i = 0; i < numPrimaryKeys; i++) { IAType keyType = (indicators == null || indicators.get(i) == 0) ? recordType.getSubFieldType(partitioningKeys.get(i)) : metaType.getSubFieldType(partitioningKeys.get(i)); cmpFactories[i] = cmpFactoryProvider.getBinaryComparatorFactory(keyType, true); } return cmpFactories; }
/** * Gets the hash function factories for the primary key fields of this dataset. * * @param metadataProvider, * the metadata provider. * @return the hash function factories for the primary key fields of this dataset. * @throws AlgebricksException */ public IBinaryHashFunctionFactory[] getPrimaryHashFunctionFactories(MetadataProvider metadataProvider) throws AlgebricksException { ARecordType recordType = (ARecordType) metadataProvider.findType(this); ARecordType metaType = (ARecordType) metadataProvider.findMetaType(this); List<List<String>> partitioningKeys = getPrimaryKeys(); int numPrimaryKeys = partitioningKeys.size(); IBinaryHashFunctionFactory[] hashFuncFactories = new IBinaryHashFunctionFactory[numPrimaryKeys]; List<Integer> indicators = null; if (hasMetaPart()) { indicators = ((InternalDatasetDetails) getDatasetDetails()).getKeySourceIndicator(); } for (int i = 0; i < numPrimaryKeys; i++) { IAType keyType = (indicators == null || indicators.get(i) == 0) ? recordType.getSubFieldType(partitioningKeys.get(i)) : metaType.getSubFieldType(partitioningKeys.get(i)); hashFuncFactories[i] = BinaryHashFunctionFactoryProvider.INSTANCE.getBinaryHashFunctionFactory(keyType); } return hashFuncFactories; }
int numPrimaryKeys = partitioningKeys.size(); ISerializerDeserializer[] primaryRecFields = new ISerializerDeserializer[numPrimaryKeys + 1 + (hasMetaPart() ? 1 : 0)]; ITypeTraits[] primaryTypeTraits = new ITypeTraits[numPrimaryKeys + 1 + (hasMetaPart() ? 1 : 0)]; ISerializerDeserializerProvider serdeProvider = metadataProvider.getDataFormat().getSerdeProvider(); List<Integer> indicators = null; if (hasMetaPart()) { indicators = ((InternalDatasetDetails) getDatasetDetails()).getKeySourceIndicator(); if (hasMetaPart()) {
@Override protected AlgebricksMetaOperatorDescriptor createCastOp(JobSpecification spec, DatasetType dsType, boolean strictCast) throws AlgebricksException { int[] outColumns = new int[1]; // tags(2) + primary keys + record + meta part(?) int[] projectionList = new int[NUM_TAG_FIELDS + (dataset.hasMetaPart() ? 2 : 1) + numPrimaryKeys]; int recordIdx = NUM_TAG_FIELDS + numPrimaryKeys; //here we only consider internal dataset assert dsType == DatasetType.INTERNAL; outColumns[0] = NUM_TAG_FIELDS + numPrimaryKeys; int projCount = 0; for (int i = 0; i < NUM_TAG_FIELDS; i++) { projectionList[projCount++] = i; } //set primary keys and the record for (int i = 0; i <= numPrimaryKeys; i++) { projectionList[projCount++] = NUM_TAG_FIELDS + i; } if (dataset.hasMetaPart()) { projectionList[NUM_TAG_FIELDS + numPrimaryKeys + 1] = NUM_TAG_FIELDS + numPrimaryKeys + 1; } IScalarEvaluatorFactory[] castEvalFact = new IScalarEvaluatorFactory[] { new ColumnAccessEvalFactory(recordIdx) }; IScalarEvaluatorFactory[] sefs = new IScalarEvaluatorFactory[1]; sefs[0] = createCastFunction(strictCast).createEvaluatorFactory(castEvalFact); AssignRuntimeFactory castAssign = new AssignRuntimeFactory(outColumns, sefs, projectionList); castAssign.setSourceLocation(sourceLoc); return new AlgebricksMetaOperatorDescriptor(spec, 1, 1, new IPushRuntimeFactory[] { castAssign }, new RecordDescriptor[] { getTaggedRecordDescriptor(enforcedRecDesc) }); }
ARecordType metaType = null; if (dataset.hasMetaPart()) { String metaTypeName = FeedUtils.getFeedMetaTypeName(sourceFeed.getConfiguration()); if (metaTypeName == null) { pkTypes = ((InternalDatasetDetails) dataset.getDatasetDetails()).getPrimaryKeyType(); partitioningKeys = ((InternalDatasetDetails) dataset.getDatasetDetails()).getPartitioningKey(); if (dataset.hasMetaPart()) { keySourceIndicator = ((InternalDatasetDetails) dataset.getDatasetDetails()).getKeySourceIndicator();
public boolean allow(ILogicalOperator topOp, byte operation) { if (!hasMetaPart()) { return true; } if (topOp.getInputs().get(0).getValue().getOperatorTag() != LogicalOperatorTag.ASSIGN) { return false; } ILogicalOperator op = topOp.getInputs().get(0).getValue(); while ((!op.getInputs().isEmpty()) && op.getInputs().get(0).getValue().getOperatorTag() != LogicalOperatorTag.UNNEST) { op = op.getInputs().get(0).getValue(); } if (op.getInputs().isEmpty()) { return false; } UnnestOperator unnestOp = (UnnestOperator) op.getInputs().get(0).getValue(); if (unnestOp.getExpressionRef().getValue().getExpressionTag() != LogicalExpressionTag.FUNCTION_CALL) { return false; } AbstractFunctionCallExpression functionCall = (AbstractFunctionCallExpression) unnestOp.getExpressionRef().getValue(); if (functionCall.getFunctionIdentifier() != BuiltinFunctions.FEED_COLLECT) { return false; } return operation == DatasetUtil.OP_UPSERT; }
if (dataset.hasMetaPart()) { metaItemType = (ARecordType) metadataProvider.findMetaType(dataset);
subTree.getDataset().getDatasetName()); List<LogicalVariable> datasetVars = subTree.getDataSourceVariables(); if (subTree.getDataset().hasMetaPart()) { datasetMetaVar = datasetVars.get(datasetVars.size() - 1);
private ILogicalOperator translateDelete(DatasetDataSource targetDatasource, Mutable<ILogicalExpression> varRef, List<Mutable<ILogicalExpression>> varRefsForLoading, List<Mutable<ILogicalExpression>> additionalFilteringExpressions, ILogicalOperator assign, ICompiledDmlStatement stmt) throws AlgebricksException { SourceLocation sourceLoc = stmt.getSourceLocation(); if (targetDatasource.getDataset().hasMetaPart()) { throw new CompilationException(ErrorCode.COMPILATION_ERROR, sourceLoc, targetDatasource.getDataset().getDatasetName() + ": delete from dataset is not supported on Datasets with Meta records"); } InsertDeleteUpsertOperator deleteOp = new InsertDeleteUpsertOperator(targetDatasource, varRef, varRefsForLoading, InsertDeleteUpsertOperator.Kind.DELETE, false); deleteOp.setAdditionalFilteringExpressions(additionalFilteringExpressions); deleteOp.getInputs().add(new MutableObject<>(assign)); deleteOp.setSourceLocation(sourceLoc); DelegateOperator leafOperator = new DelegateOperator(new CommitOperator(true)); leafOperator.getInputs().add(new MutableObject<>(deleteOp)); leafOperator.setSourceLocation(sourceLoc); return leafOperator; }
private ILogicalOperator translateInsert(DatasetDataSource targetDatasource, Mutable<ILogicalExpression> varRef, List<Mutable<ILogicalExpression>> varRefsForLoading, List<Mutable<ILogicalExpression>> additionalFilteringExpressions, ILogicalOperator assign, ICompiledDmlStatement stmt) throws AlgebricksException { SourceLocation sourceLoc = stmt.getSourceLocation(); if (targetDatasource.getDataset().hasMetaPart()) { throw new CompilationException(ErrorCode.COMPILATION_ERROR, sourceLoc, targetDatasource.getDataset().getDatasetName() + ": insert into dataset is not supported on Datasets with Meta records"); } // Adds the insert operator. InsertDeleteUpsertOperator insertOp = new InsertDeleteUpsertOperator(targetDatasource, varRef, varRefsForLoading, InsertDeleteUpsertOperator.Kind.INSERT, false); insertOp.setAdditionalFilteringExpressions(additionalFilteringExpressions); insertOp.getInputs().add(new MutableObject<>(assign)); insertOp.setSourceLocation(sourceLoc); // Adds the commit operator. CompiledInsertStatement compiledInsert = (CompiledInsertStatement) stmt; Expression returnExpression = compiledInsert.getReturnExpression(); DelegateOperator rootOperator = new DelegateOperator(new CommitOperator(returnExpression == null)); rootOperator.getInputs().add(new MutableObject<>(insertOp)); rootOperator.setSourceLocation(sourceLoc); // Compiles the return expression. return processReturningExpression(rootOperator, insertOp, compiledInsert); }
private RecordDescriptor getUpsertOutRecDesc(RecordDescriptor inputRecordDesc, Dataset dataset, int numFilterFields, ARecordType itemType, ARecordType metaItemType) throws Exception { ITypeTraits[] outputTypeTraits = new ITypeTraits[inputRecordDesc.getFieldCount() + (dataset.hasMetaPart() ? 2 : 1) + numFilterFields]; ISerializerDeserializer<?>[] outputSerDes = new ISerializerDeserializer[inputRecordDesc.getFieldCount() + (dataset.hasMetaPart() ? 2 : 1) + numFilterFields]; f++; if (dataset.hasMetaPart()) { outputSerDes[f] = FormatUtils.getDefaultFormat().getSerdeProvider().getSerializerDeserializer(metaItemType); outputTypeTraits[f] = FormatUtils.getDefaultFormat().getTypeTraitProvider().getTypeTrait(metaItemType);
private void writeMetaPart(Dataset dataset) throws HyracksDataException { if (dataset.hasMetaPart()) { // write open field 1, the meta item type Dataverse name. fieldName.reset(); aString.setValue(MetadataRecordTypes.FIELD_NAME_METADATA_DATAVERSE); stringSerde.serialize(aString, fieldName.getDataOutput()); fieldValue.reset(); aString.setValue(dataset.getMetaItemTypeDataverseName()); stringSerde.serialize(aString, fieldValue.getDataOutput()); recordBuilder.addField(fieldName, fieldValue); // write open field 2, the meta item type name. fieldName.reset(); aString.setValue(MetadataRecordTypes.FIELD_NAME_METATYPE_NAME); stringSerde.serialize(aString, fieldName.getDataOutput()); fieldValue.reset(); aString.setValue(dataset.getMetaItemTypeName()); stringSerde.serialize(aString, fieldValue.getDataOutput()); recordBuilder.addField(fieldName, fieldValue); } }