@Override public RESULT visitCreateIndex(CONTEXT context, LogicalPlan plan, LogicalPlan.QueryBlock block, CreateIndexNode node, Stack<LogicalNode> stack) throws TajoException { RESULT result = null; stack.push(node); result = visit(context, plan, block, node.getChild(), stack); stack.pop(); return result; }
private void checkIndexExistence(final QueryContext queryContext, final CreateIndexNode createIndexNode) throws DuplicateIndexException { String databaseName, simpleIndexName, qualifiedIndexName; if (IdentifierUtil.isFQTableName(createIndexNode.getIndexName())) { String[] splits = IdentifierUtil.splitFQTableName(createIndexNode.getIndexName()); databaseName = splits[0]; simpleIndexName = splits[1]; qualifiedIndexName = createIndexNode.getIndexName(); } else { databaseName = queryContext.getCurrentDatabase(); simpleIndexName = createIndexNode.getIndexName(); qualifiedIndexName = IdentifierUtil.buildFQName(databaseName, simpleIndexName); } if (catalog.existIndexByName(databaseName, simpleIndexName)) { throw new DuplicateIndexException(qualifiedIndexName); } }
@Override public LogicalNode visitCreateIndex(GlobalPlanContext context, LogicalPlan plan, LogicalPlan.QueryBlock queryBlock, CreateIndexNode node, Stack<LogicalNode> stack) throws TajoException { LogicalNode child = super.visitCreateIndex(context, plan, queryBlock, node, stack); // Don't separate execution block. CreateIndex is pushed to the first execution block. ExecutionBlock childBlock = context.execBlockMap.remove(child.getPID()); node.setChild(childBlock.getPlan()); childBlock.setPlan(node); context.execBlockMap.put(node.getPID(), childBlock); return node; } }
public StoreIndexExec(final TaskAttemptContext context, final CreateIndexNode logicalPlan, final PhysicalExec child) { super(context, logicalPlan.getInSchema(), logicalPlan.getOutSchema(), child); this.logicalPlan = logicalPlan; }
@Override public LogicalNode visitCreateIndex(SerializeContext context, LogicalPlan plan, LogicalPlan.QueryBlock block, CreateIndexNode node, Stack<LogicalNode> stack) throws TajoException { super.visitCreateIndex(context, plan, block, node, new Stack<>()); PlanProto.CreateIndexNode.Builder createIndexBuilder = PlanProto.CreateIndexNode.newBuilder(); int [] childIds = registerGetChildIds(context, node); createIndexBuilder.setChildSeq(childIds[0]); createIndexBuilder.setIndexName(node.getIndexName()); createIndexBuilder.setIndexMethod(node.getIndexMethod()); createIndexBuilder.setIndexPath(node.getIndexPath().toString()); for (SortSpec sortSpec : node.getKeySortSpecs()) { createIndexBuilder.addKeySortSpecs(sortSpec.getProto()); } createIndexBuilder.setTargetRelationSchema(node.getTargetRelationSchema().getProto()); createIndexBuilder.setIsUnique(node.isUnique()); createIndexBuilder.setIsClustered(node.isClustered()); if (node.hasOptions()) { createIndexBuilder.setIndexProperties(node.getOptions().getProto()); } createIndexBuilder.setIsExternal(node.isExternal()); PlanProto.LogicalNode.Builder nodeBuilder = createNodeBuilder(context, node); nodeBuilder.setCreateIndex(createIndexBuilder); context.treeBuilder.addNodes(nodeBuilder); return node; }
public void createIndex(final QueryContext queryContext, final CreateIndexNode createIndexNode) throws DuplicateIndexException, UndefinedTableException, UndefinedDatabaseException { String databaseName, simpleIndexName, qualifiedIndexName; if (CatalogUtil.isFQTableName(createIndexNode.getIndexName())) { String[] splits = CatalogUtil.splitFQTableName(createIndexNode.getIndexName()); databaseName = splits[0]; simpleIndexName = splits[1]; qualifiedIndexName = createIndexNode.getIndexName(); } else { databaseName = queryContext.getCurrentDatabase(); simpleIndexName = createIndexNode.getIndexName(); qualifiedIndexName = CatalogUtil.buildFQName(databaseName, simpleIndexName); } if (catalog.existIndexByName(databaseName, simpleIndexName)) { throw new DuplicateIndexException(simpleIndexName); } ScanNode scanNode = PlannerUtil.findTopNode(createIndexNode, NodeType.SCAN); if (scanNode == null) { throw new InternalError("Cannot find the table of the relation"); } IndexDesc indexDesc = new IndexDesc(databaseName, CatalogUtil.extractSimpleName(scanNode.getTableName()), simpleIndexName, createIndexNode.getIndexPath(), createIndexNode.getKeySortSpecs(), createIndexNode.getIndexMethod(), createIndexNode.isUnique(), false, scanNode.getLogicalSchema()); catalog.createIndex(indexDesc); LOG.info("Index " + qualifiedIndexName + " is created for the table " + scanNode.getTableName() + "."); }
private static CreateIndexNode convertCreateIndex(Map<Integer, LogicalNode> nodeMap, PlanProto.LogicalNode protoNode) { CreateIndexNode createIndex = new CreateIndexNode(protoNode.getNodeId()); PlanProto.CreateIndexNode createIndexProto = protoNode.getCreateIndex(); createIndex.setIndexName(createIndexProto.getIndexName()); createIndex.setIndexMethod(createIndexProto.getIndexMethod()); try { createIndex.setIndexPath(new URI(createIndexProto.getIndexPath())); } catch (URISyntaxException e) { e.printStackTrace(); } SortSpec[] keySortSpecs = new SortSpec[createIndexProto.getKeySortSpecsCount()]; for (int i = 0; i < keySortSpecs.length; i++) { keySortSpecs[i] = new SortSpec(createIndexProto.getKeySortSpecs(i)); } createIndex.setKeySortSpecs(new Schema(createIndexProto.getTargetRelationSchema()), keySortSpecs); createIndex.setUnique(createIndexProto.getIsUnique()); createIndex.setClustered(createIndexProto.getIsClustered()); if (createIndexProto.hasIndexProperties()) { createIndex.setOptions(new KeyValueSet(createIndexProto.getIndexProperties())); } createIndex.setChild(nodeMap.get(createIndexProto.getChildSeq())); createIndex.setInSchema(convertSchema(protoNode.getInSchema())); createIndex.setOutSchema(convertSchema(protoNode.getOutSchema())); createIndex.setExternal(createIndexProto.getIsExternal()); return createIndex; }
CreateIndexNode createIndexNode = block.getNodeFromExpr(createIndex); if (IdentifierUtil.isFQTableName(createIndex.getIndexName())) { createIndexNode.setIndexName(createIndex.getIndexName()); } else { createIndexNode.setIndexName( IdentifierUtil.buildFQName(context.queryContext.get(SessionVars.CURRENT_DATABASE), createIndex.getIndexName())); createIndexNode.setUnique(createIndex.isUnique()); Sort.SortSpec[] sortSpecs = createIndex.getSortSpecs(); int sortKeyNum = sortSpecs.length; createIndexNode.setExternal(createIndex.isExternal()); createIndexNode.setKeySortSpecs(relationNode.getLogicalSchema(), annotateSortSpecs(block, referNames, sortSpecs)); createIndexNode.setIndexMethod(IndexMethod.valueOf(createIndex.getMethodSpec().getName().toUpperCase())); if (createIndex.isExternal()) { createIndexNode.setIndexPath(new Path(createIndex.getIndexPath()).toUri()); } else { createIndexNode.setIndexPath( getIndexPath(context, context.queryContext.get(SessionVars.CURRENT_DATABASE), createIndex.getIndexName())); KeyValueSet keyValueSet = new KeyValueSet(); keyValueSet.putAll(createIndex.getParams()); createIndexNode.setOptions(keyValueSet); createIndexNode.setChild(child); return createIndexNode;
@Override public void init() throws IOException { super.init(); SortSpec[] sortSpecs = logicalPlan.getKeySortSpecs(); indexKeys = new int[sortSpecs.length]; keySchema = PlannerUtil.sortSpecsToSchema(sortSpecs); Column col; for (int i = 0 ; i < sortSpecs.length; i++) { col = sortSpecs[i].getSortKey(); indexKeys[i] = inSchema.getColumnId(col.getQualifiedName()); } // TODO: this line should be improved to allow multiple scan executors. ScanExec scanExec = PhysicalPlanUtil.findExecutor(this, ScanExec.class); if (scanExec == null) { throw new TajoInternalError("Cannot find scan executors."); } TajoConf conf = context.getConf(); Path indexPath = new Path(logicalPlan.getIndexPath().toString(), IndexExecutorUtil.getIndexFileName(scanExec.getFragments()[0])); // TODO: Create factory using reflection BSTIndex bst = new BSTIndex(conf); this.comparator = new BaseTupleComparator(keySchema, sortSpecs); this.indexWriter = bst.getIndexWriter(indexPath, BSTIndex.TWO_LEVEL_INDEX, keySchema, comparator); this.indexWriter.setLoadNum(100); this.indexWriter.init(); }
case CREATE_INDEX: CreateIndexNode createIndexNode = (CreateIndexNode) parentNode; createIndexNode.setChild(child); createIndexNode.setInSchema(child.getOutSchema()); break; default:
@Override public LogicalNode visitCreateIndex(SerializeContext context, LogicalPlan plan, LogicalPlan.QueryBlock block, CreateIndexNode node, Stack<LogicalNode> stack) throws TajoException { super.visitCreateIndex(context, plan, block, node, new Stack<LogicalNode>()); PlanProto.CreateIndexNode.Builder createIndexBuilder = PlanProto.CreateIndexNode.newBuilder(); int [] childIds = registerGetChildIds(context, node); createIndexBuilder.setChildSeq(childIds[0]); createIndexBuilder.setIndexName(node.getIndexName()); createIndexBuilder.setIndexMethod(node.getIndexMethod()); createIndexBuilder.setIndexPath(node.getIndexPath().toString()); for (SortSpec sortSpec : node.getKeySortSpecs()) { createIndexBuilder.addKeySortSpecs(sortSpec.getProto()); } createIndexBuilder.setTargetRelationSchema(node.getTargetRelationSchema().getProto()); createIndexBuilder.setIsUnique(node.isUnique()); createIndexBuilder.setIsClustered(node.isClustered()); if (node.hasOptions()) { createIndexBuilder.setIndexProperties(node.getOptions().getProto()); } createIndexBuilder.setIsExternal(node.isExternal()); PlanProto.LogicalNode.Builder nodeBuilder = createNodeBuilder(context, node); nodeBuilder.setCreateIndex(createIndexBuilder); context.treeBuilder.addNodes(nodeBuilder); return node; }
public void createIndex(final QueryContext queryContext, final CreateIndexNode createIndexNode) throws DuplicateIndexException, UndefinedTableException, UndefinedDatabaseException { String databaseName, simpleIndexName, qualifiedIndexName; if (IdentifierUtil.isFQTableName(createIndexNode.getIndexName())) { String[] splits = IdentifierUtil.splitFQTableName(createIndexNode.getIndexName()); databaseName = splits[0]; simpleIndexName = splits[1]; qualifiedIndexName = createIndexNode.getIndexName(); } else { databaseName = queryContext.getCurrentDatabase(); simpleIndexName = createIndexNode.getIndexName(); qualifiedIndexName = IdentifierUtil.buildFQName(databaseName, simpleIndexName); } if (catalog.existIndexByName(databaseName, simpleIndexName)) { throw new DuplicateIndexException(simpleIndexName); } ScanNode scanNode = PlannerUtil.findTopNode(createIndexNode, NodeType.SCAN); if (scanNode == null) { throw new InternalError("Cannot find the table of the relation"); } IndexDesc indexDesc = new IndexDesc(databaseName, IdentifierUtil.extractSimpleName(scanNode.getTableName()), simpleIndexName, createIndexNode.getIndexPath(), createIndexNode.getKeySortSpecs(), createIndexNode.getIndexMethod(), createIndexNode.isUnique(), false, scanNode.getLogicalSchema()); catalog.createIndex(indexDesc); LOG.info("Index " + qualifiedIndexName + " is created for the table " + scanNode.getTableName() + "."); }
private static CreateIndexNode convertCreateIndex(Map<Integer, LogicalNode> nodeMap, PlanProto.LogicalNode protoNode) { CreateIndexNode createIndex = new CreateIndexNode(protoNode.getNodeId()); PlanProto.CreateIndexNode createIndexProto = protoNode.getCreateIndex(); createIndex.setIndexName(createIndexProto.getIndexName()); createIndex.setIndexMethod(createIndexProto.getIndexMethod()); try { createIndex.setIndexPath(new URI(createIndexProto.getIndexPath())); } catch (URISyntaxException e) { e.printStackTrace(); } SortSpec[] keySortSpecs = new SortSpec[createIndexProto.getKeySortSpecsCount()]; for (int i = 0; i < keySortSpecs.length; i++) { keySortSpecs[i] = new SortSpec(createIndexProto.getKeySortSpecs(i)); } createIndex.setKeySortSpecs(SchemaFactory.newV1(createIndexProto.getTargetRelationSchema()), keySortSpecs); createIndex.setUnique(createIndexProto.getIsUnique()); createIndex.setClustered(createIndexProto.getIsClustered()); if (createIndexProto.hasIndexProperties()) { createIndex.setOptions(new KeyValueSet(createIndexProto.getIndexProperties())); } createIndex.setChild(nodeMap.get(createIndexProto.getChildSeq())); createIndex.setInSchema(convertSchema(protoNode.getInSchema())); createIndex.setOutSchema(convertSchema(protoNode.getOutSchema())); createIndex.setExternal(createIndexProto.getIsExternal()); return createIndex; }
CreateIndexNode createIndexNode = block.getNodeFromExpr(createIndex); if (CatalogUtil.isFQTableName(createIndex.getIndexName())) { createIndexNode.setIndexName(createIndex.getIndexName()); } else { createIndexNode.setIndexName( CatalogUtil.buildFQName(context.queryContext.get(SessionVars.CURRENT_DATABASE), createIndex.getIndexName())); createIndexNode.setUnique(createIndex.isUnique()); Sort.SortSpec[] sortSpecs = createIndex.getSortSpecs(); int sortKeyNum = sortSpecs.length; createIndexNode.setExternal(createIndex.isExternal()); createIndexNode.setKeySortSpecs(relationNode.getLogicalSchema(), annotateSortSpecs(block, referNames, sortSpecs)); createIndexNode.setIndexMethod(IndexMethod.valueOf(createIndex.getMethodSpec().getName().toUpperCase())); if (createIndex.isExternal()) { createIndexNode.setIndexPath(new Path(createIndex.getIndexPath()).toUri()); } else { createIndexNode.setIndexPath( getIndexPath(context, context.queryContext.get(SessionVars.CURRENT_DATABASE), createIndex.getIndexName())); KeyValueSet keyValueSet = new KeyValueSet(); keyValueSet.putAll(createIndex.getParams()); createIndexNode.setOptions(keyValueSet); createIndexNode.setChild(child); return createIndexNode;
@Override public void init() throws IOException { super.init(); SortSpec[] sortSpecs = logicalPlan.getKeySortSpecs(); indexKeys = new int[sortSpecs.length]; keySchema = PlannerUtil.sortSpecsToSchema(sortSpecs); Column col; for (int i = 0 ; i < sortSpecs.length; i++) { col = sortSpecs[i].getSortKey(); indexKeys[i] = inSchema.getColumnId(col.getQualifiedName()); } // TODO: this line should be improved to allow multiple scan executors. ScanExec scanExec = PhysicalPlanUtil.findExecutor(this, ScanExec.class); if (scanExec == null) { throw new TajoInternalError("Cannot find scan executors."); } TajoConf conf = context.getConf(); Path indexPath = new Path(logicalPlan.getIndexPath().toString(), IndexExecutorUtil.getIndexFileName(conf, scanExec.getFragments()[0])); // TODO: Create factory using reflection BSTIndex bst = new BSTIndex(conf); this.comparator = new BaseTupleComparator(keySchema, sortSpecs); this.indexWriter = bst.getIndexWriter(indexPath, BSTIndex.TWO_LEVEL_INDEX, keySchema, comparator); this.indexWriter.setLoadNum(100); this.indexWriter.init(); }
case CREATE_INDEX: CreateIndexNode createIndexNode = (CreateIndexNode) parentNode; createIndexNode.setChild(child); createIndexNode.setInSchema(child.getOutSchema()); break; default:
public StoreIndexExec(final TaskAttemptContext context, final CreateIndexNode logicalPlan, final PhysicalExec child) { super(context, logicalPlan.getInSchema(), logicalPlan.getOutSchema(), child); this.logicalPlan = logicalPlan; }
@Override public void execute(QueryMaster.QueryMasterContext context, QueryContext queryContext, Query query, ExecutionBlockId finalExecBlockId, Path finalOutputDir) throws Exception { CatalogService catalog = context.getWorkerContext().getCatalog(); Stage lastStage = query.getStage(finalExecBlockId); CreateIndexNode createIndexNode = (CreateIndexNode) lastStage.getBlock().getPlan(); String databaseName, simpleIndexName, qualifiedIndexName; if (CatalogUtil.isFQTableName(createIndexNode.getIndexName())) { String [] splits = CatalogUtil.splitFQTableName(createIndexNode.getIndexName()); databaseName = splits[0]; simpleIndexName = splits[1]; qualifiedIndexName = createIndexNode.getIndexName(); } else { databaseName = queryContext.getCurrentDatabase(); simpleIndexName = createIndexNode.getIndexName(); qualifiedIndexName = CatalogUtil.buildFQName(databaseName, simpleIndexName); } ScanNode scanNode = PlannerUtil.findTopNode(createIndexNode, NodeType.SCAN); if (scanNode == null) { throw new IOException("Cannot find the table of the relation"); } IndexDesc indexDesc = new IndexDesc(databaseName, CatalogUtil.extractSimpleName(scanNode.getTableName()), simpleIndexName, createIndexNode.getIndexPath(), createIndexNode.getKeySortSpecs(), createIndexNode.getIndexMethod(), createIndexNode.isUnique(), false, scanNode.getLogicalSchema()); catalog.createIndex(indexDesc); LOG.info("Index " + qualifiedIndexName + " is created for the table " + scanNode.getTableName() + "."); } }
@Override public RESULT visitCreateIndex(CONTEXT context, LogicalPlan plan, LogicalPlan.QueryBlock block, CreateIndexNode node, Stack<LogicalNode> stack) throws TajoException { RESULT result = null; stack.push(node); result = visit(context, plan, block, node.getChild(), stack); stack.pop(); return result; }
@Override public LogicalNode visitCreateIndex(GlobalPlanContext context, LogicalPlan plan, LogicalPlan.QueryBlock queryBlock, CreateIndexNode node, Stack<LogicalNode> stack) throws TajoException { LogicalNode child = super.visitCreateIndex(context, plan, queryBlock, node, stack); // Don't separate execution block. CreateIndex is pushed to the first execution block. ExecutionBlock childBlock = context.execBlockMap.remove(child.getPID()); node.setChild(childBlock.getPlan()); childBlock.setPlan(node); context.execBlockMap.put(node.getPID(), childBlock); return node; } }