@Override public ConnectorSplitSource getSplits(ConnectorTransactionHandle transaction, ConnectorSession session, ConnectorTableLayoutHandle layout, SplitSchedulingStrategy splitSchedulingStrategy) { JmxTableLayoutHandle jmxLayout = (JmxTableLayoutHandle) layout; JmxTableHandle tableHandle = jmxLayout.getTable(); TupleDomain<ColumnHandle> predicate = jmxLayout.getConstraint(); //TODO is there a better way to get the node column? Optional<JmxColumnHandle> nodeColumnHandle = tableHandle.getColumnHandles().stream() .filter(jmxColumnHandle -> jmxColumnHandle.getColumnName().equals(NODE_COLUMN_NAME)) .findFirst(); checkState(nodeColumnHandle.isPresent(), "Failed to find %s column", NODE_COLUMN_NAME); List<ConnectorSplit> splits = nodeManager.getAllNodes().stream() .filter(node -> { NullableValue value = NullableValue.of(createUnboundedVarcharType(), utf8Slice(node.getNodeIdentifier())); return predicate.overlaps(fromFixedValues(ImmutableMap.of(nodeColumnHandle.get(), value))); }) .map(node -> new JmxSplit(tableHandle, ImmutableList.of(node.getHostAndPort()))) .collect(toList()); return new FixedSplitSource(splits); } }
case TEXT: case VARCHAR: return NullableValue.of(nativeType, utf8Slice(row.getString(i))); case INT: return NullableValue.of(nativeType, (long) row.getInt(i)); case BIGINT: case COUNTER: return NullableValue.of(nativeType, row.getLong(i)); case BOOLEAN: return NullableValue.of(nativeType, row.getBool(i)); case DOUBLE: return NullableValue.of(nativeType, row.getDouble(i)); case FLOAT: return NullableValue.of(nativeType, (long) floatToRawIntBits(row.getFloat(i))); case DECIMAL: return NullableValue.of(nativeType, row.getDecimal(i).doubleValue()); case UUID: case TIMEUUID: return NullableValue.of(nativeType, utf8Slice(row.getUUID(i).toString())); case TIMESTAMP: return NullableValue.of(nativeType, row.getTimestamp(i).getTime()); case INET: return NullableValue.of(nativeType, utf8Slice(toAddrString(row.getInet(i)))); case VARINT: return NullableValue.of(nativeType, utf8Slice(row.getVarint(i).toString())); case BLOB: case CUSTOM: return NullableValue.of(nativeType, wrappedBuffer(row.getBytesUnsafe(i))); case SET:
public static NullableValue getColumnValueForPartitionKey(Row row, int i, CassandraType cassandraType, List<CassandraType> typeArguments) { Type nativeType = cassandraType.getNativeType(); if (row.isNull(i)) { return NullableValue.asNull(nativeType); } switch (cassandraType) { case ASCII: case TEXT: case VARCHAR: return NullableValue.of(nativeType, utf8Slice(row.getString(i))); case UUID: case TIMEUUID: return NullableValue.of(nativeType, utf8Slice(row.getUUID(i).toString())); default: return getColumnValue(row, i, cassandraType, typeArguments); } }
return NullableValue.of(decimalType, 0L); return NullableValue.of(decimalType, shortDecimalPartitionKey(value, decimalType, partitionName)); return NullableValue.of(decimalType, Decimals.encodeUnscaledValue(BigInteger.ZERO)); return NullableValue.of(decimalType, longDecimalPartitionKey(value, decimalType, partitionName)); return NullableValue.of(BOOLEAN, false); return NullableValue.of(BOOLEAN, booleanPartitionKey(value, partitionName)); return NullableValue.of(TINYINT, 0L); return NullableValue.of(TINYINT, tinyintPartitionKey(value, partitionName)); return NullableValue.of(SMALLINT, 0L); return NullableValue.of(SMALLINT, smallintPartitionKey(value, partitionName)); return NullableValue.of(INTEGER, 0L); return NullableValue.of(INTEGER, integerPartitionKey(value, partitionName)); return NullableValue.of(BIGINT, 0L); return NullableValue.of(BIGINT, bigintPartitionKey(value, partitionName)); return NullableValue.of(DATE, datePartitionKey(value, partitionName));
@Test public void testSingleValuesMapToDomain() { assertEquals( TupleDomain.fromFixedValues( ImmutableMap.<ColumnHandle, NullableValue>builder() .put(A, NullableValue.of(BIGINT, 1L)) .put(B, NullableValue.of(VARCHAR, utf8Slice("value"))) .put(C, NullableValue.of(DOUBLE, 0.01)) .put(D, NullableValue.asNull(BOOLEAN)) .build()), TupleDomain.withColumnDomains(ImmutableMap.<ColumnHandle, Domain>builder() .put(A, Domain.singleValue(BIGINT, 1L)) .put(B, Domain.singleValue(VARCHAR, utf8Slice("value"))) .put(C, Domain.singleValue(DOUBLE, 0.01)) .put(D, Domain.onlyNull(BOOLEAN)) .build())); }
private boolean rowMatchesPredicate() { if (predicate.isAll()) { return true; } if (predicate.isNone()) { return false; } Map<ColumnHandle, NullableValue> rowMap = predicate.getDomains().get().keySet().stream() .collect(toImmutableMap( column -> column, column -> { TpchColumnHandle tpchColumnHandle = (TpchColumnHandle) column; Type type = tpchColumnHandle.getType(); TpchColumn tpchColumn = table.getColumn(tpchColumnHandle.getColumnName()); return NullableValue.of(type, getPrestoObject(tpchColumn, type)); })); TupleDomain<ColumnHandle> rowTupleDomain = TupleDomain.fromFixedValues(rowMap); return predicate.contains(rowTupleDomain); }
@Test public void testBucketedTableDoubleFloat() throws Exception { try (Transaction transaction = newTransaction()) { ConnectorMetadata metadata = transaction.getMetadata(); ConnectorSession session = newSession(); ConnectorTableHandle tableHandle = getTableHandle(metadata, tableBucketedDoubleFloat); List<ColumnHandle> columnHandles = ImmutableList.copyOf(metadata.getColumnHandles(session, tableHandle).values()); Map<String, Integer> columnIndex = indexColumns(columnHandles); assertTableIsBucketed(tableHandle); ImmutableMap<ColumnHandle, NullableValue> bindings = ImmutableMap.<ColumnHandle, NullableValue>builder() .put(columnHandles.get(columnIndex.get("t_float")), NullableValue.of(REAL, (long) floatToRawIntBits(87.1f))) .put(columnHandles.get(columnIndex.get("t_double")), NullableValue.of(DOUBLE, 88.2)) .build(); // floats and doubles are not supported, so we should see all splits MaterializedResult result = readTable(transaction, tableHandle, columnHandles, session, TupleDomain.fromFixedValues(bindings), OptionalInt.of(32), Optional.empty()); assertEquals(result.getRowCount(), 100); } }
@Test public void testCreateMixedPredicate() { ImmutableList.Builder<HivePartition> partitions = ImmutableList.builder(); for (int i = 0; i < 5; i++) { partitions.add(new HivePartition( new SchemaTableName("test", "test"), Integer.toString(i), ImmutableMap.of(TEST_COLUMN_HANDLE, NullableValue.of(VarcharType.VARCHAR, Slices.utf8Slice(Integer.toString(i)))))); } partitions.add(new HivePartition( new SchemaTableName("test", "test"), "null", ImmutableMap.of(TEST_COLUMN_HANDLE, NullableValue.asNull(VarcharType.VARCHAR)))); createPredicate(ImmutableList.of(TEST_COLUMN_HANDLE), partitions.build()); } }
@Test public void testPredicatePushdown() throws Exception { for (Node node : nodes) { String nodeIdentifier = node.getNodeIdentifier(); TupleDomain<ColumnHandle> nodeTupleDomain = TupleDomain.fromFixedValues(ImmutableMap.of(columnHandle, NullableValue.of(createUnboundedVarcharType(), utf8Slice(nodeIdentifier)))); ConnectorTableLayoutHandle layout = new JmxTableLayoutHandle(tableHandle, nodeTupleDomain); ConnectorSplitSource splitSource = splitManager.getSplits(JmxTransactionHandle.INSTANCE, SESSION, layout, UNGROUPED_SCHEDULING); List<ConnectorSplit> allSplits = getAllSplits(splitSource); assertEquals(allSplits.size(), 1); assertEquals(allSplits.get(0).getAddresses().size(), 1); assertEquals(allSplits.get(0).getAddresses().get(0).getHostText(), nodeIdentifier); } }
@Test public void testExtractFixedValues() { assertEquals( TupleDomain.extractFixedValues(TupleDomain.withColumnDomains( ImmutableMap.<ColumnHandle, Domain>builder() .put(A, Domain.all(DOUBLE)) .put(B, Domain.singleValue(VARCHAR, utf8Slice("value"))) .put(C, Domain.onlyNull(BIGINT)) .put(D, Domain.create(ValueSet.ofRanges(Range.equal(BIGINT, 1L)), true)) .build())).get(), ImmutableMap.of( B, NullableValue.of(VARCHAR, utf8Slice("value")), C, NullableValue.asNull(BIGINT))); }
.put(columnHandles.get(columnIndex.get("t_string")), NullableValue.of(createUnboundedVarcharType(), utf8Slice(testString))) .put(columnHandles.get(columnIndex.get("t_bigint")), NullableValue.of(BIGINT, testBigint)) .put(columnHandles.get(columnIndex.get("t_boolean")), NullableValue.of(BOOLEAN, testBoolean)) .build();
.put(columnHandles.get(columnIndex.get("t_int")), NullableValue.of(INTEGER, (long) testInt)) .put(columnHandles.get(columnIndex.get("t_string")), NullableValue.of(createUnboundedVarcharType(), utf8Slice(testString))) .put(columnHandles.get(columnIndex.get("t_smallint")), NullableValue.of(SMALLINT, (long) testSmallint)) .build();
@Test public void testJsonSerialization() throws Exception { TestingTypeManager typeManager = new TestingTypeManager(); TestingBlockEncodingSerde blockEncodingSerde = new TestingBlockEncodingSerde(typeManager); ObjectMapper mapper = new ObjectMapperProvider().get() .registerModule(new SimpleModule() .addDeserializer(ColumnHandle.class, new JsonDeserializer<ColumnHandle>() { @Override public ColumnHandle deserialize(JsonParser jsonParser, DeserializationContext deserializationContext) throws IOException { return new ObjectMapperProvider().get().readValue(jsonParser, TestingColumnHandle.class); } }) .addDeserializer(Type.class, new TestingTypeDeserializer(typeManager)) .addSerializer(Block.class, new TestingBlockJsonSerde.Serializer(blockEncodingSerde)) .addDeserializer(Block.class, new TestingBlockJsonSerde.Deserializer(blockEncodingSerde))); TupleDomain<ColumnHandle> tupleDomain = TupleDomain.all(); assertEquals(tupleDomain, mapper.readValue(mapper.writeValueAsString(tupleDomain), new TypeReference<TupleDomain<ColumnHandle>>() {})); tupleDomain = TupleDomain.none(); assertEquals(tupleDomain, mapper.readValue(mapper.writeValueAsString(tupleDomain), new TypeReference<TupleDomain<ColumnHandle>>() {})); tupleDomain = TupleDomain.fromFixedValues(ImmutableMap.of(A, NullableValue.of(BIGINT, 1L), B, NullableValue.asNull(VARCHAR))); assertEquals(tupleDomain, mapper.readValue(mapper.writeValueAsString(tupleDomain), new TypeReference<TupleDomain<ColumnHandle>>() {})); }
@Test(enabled = false) public void testPartitionSchemaNonCanonical() throws Exception { try (Transaction transaction = newTransaction()) { ConnectorSession session = newSession(); ConnectorMetadata metadata = transaction.getMetadata(); ConnectorTableHandle table = getTableHandle(metadata, tablePartitionSchemaChangeNonCanonical); ColumnHandle column = metadata.getColumnHandles(session, table).get("t_boolean"); assertNotNull(column); List<ConnectorTableLayoutResult> tableLayoutResults = metadata.getTableLayouts(session, table, new Constraint<>(TupleDomain.fromFixedValues(ImmutableMap.of(column, NullableValue.of(BOOLEAN, false)))), Optional.empty()); ConnectorTableLayoutHandle layoutHandle = getOnlyElement(tableLayoutResults).getTableLayout().getHandle(); assertEquals(getAllPartitions(layoutHandle).size(), 1); assertEquals(getPartitionId(getAllPartitions(layoutHandle).get(0)), "t_boolean=0"); ConnectorSplitSource splitSource = splitManager.getSplits(transaction.getTransactionHandle(), session, layoutHandle, UNGROUPED_SCHEDULING); ConnectorSplit split = getOnlyElement(getAllSplits(splitSource)); ImmutableList<ColumnHandle> columnHandles = ImmutableList.of(column); try (ConnectorPageSource ignored = pageSourceProvider.createPageSource(transaction.getTransactionHandle(), session, split, columnHandles)) { fail("expected exception"); } catch (PrestoException e) { assertEquals(e.getErrorCode(), HIVE_INVALID_PARTITION_VALUE.toErrorCode()); } } }
columnHandles, session, TupleDomain.fromFixedValues(ImmutableMap.of(bucketColumnHandle(), NullableValue.of(INTEGER, 6L))), OptionalInt.empty(), Optional.empty()); columnHandles, session, TupleDomain.fromFixedValues(ImmutableMap.of(bucketColumnHandle(), NullableValue.of(INTEGER, 6L))), OptionalInt.empty(), Optional.empty());
"ds=2012-12-29/file_format=textfile/dummy=1", ImmutableMap.<ColumnHandle, NullableValue>builder() .put(dsColumn, NullableValue.of(createUnboundedVarcharType(), utf8Slice("2012-12-29"))) .put(fileFormatColumn, NullableValue.of(createUnboundedVarcharType(), utf8Slice("textfile"))) .put(dummyColumn, NullableValue.of(INTEGER, 1L)) .build())) .add(new HivePartition(tablePartitionFormat, "ds=2012-12-29/file_format=sequencefile/dummy=2", ImmutableMap.<ColumnHandle, NullableValue>builder() .put(dsColumn, NullableValue.of(createUnboundedVarcharType(), utf8Slice("2012-12-29"))) .put(fileFormatColumn, NullableValue.of(createUnboundedVarcharType(), utf8Slice("sequencefile"))) .put(dummyColumn, NullableValue.of(INTEGER, 2L)) .build())) .add(new HivePartition(tablePartitionFormat, "ds=2012-12-29/file_format=rctext/dummy=3", ImmutableMap.<ColumnHandle, NullableValue>builder() .put(dsColumn, NullableValue.of(createUnboundedVarcharType(), utf8Slice("2012-12-29"))) .put(fileFormatColumn, NullableValue.of(createUnboundedVarcharType(), utf8Slice("rctext"))) .put(dummyColumn, NullableValue.of(INTEGER, 3L)) .build())) .add(new HivePartition(tablePartitionFormat, "ds=2012-12-29/file_format=rcbinary/dummy=4", ImmutableMap.<ColumnHandle, NullableValue>builder() .put(dsColumn, NullableValue.of(createUnboundedVarcharType(), utf8Slice("2012-12-29"))) .put(fileFormatColumn, NullableValue.of(createUnboundedVarcharType(), utf8Slice("rcbinary"))) .put(dummyColumn, NullableValue.of(INTEGER, 4L)) .build())) .build(); partitionCount = partitions.size();
TupleDomain<ColumnHandle> tupleDomain = TupleDomain.fromFixedValues(ImmutableMap.of(dsColumnHandle, NullableValue.of(createUnboundedVarcharType(), utf8Slice("2015-07-03")))); Constraint<ColumnHandle> constraint = new Constraint<>(tupleDomain, convertToPredicate(tupleDomain)); List<ConnectorTableLayoutResult> tableLayoutResults = metadata.getTableLayouts(session, tableHandle, constraint, Optional.empty());
@Test public void testBucketedTableDoubleFloat() throws Exception { try (Transaction transaction = newTransaction()) { ConnectorMetadata metadata = transaction.getMetadata(); ConnectorSession session = newSession(); ConnectorTableHandle tableHandle = getTableHandle(metadata, tableBucketedDoubleFloat); List<ColumnHandle> columnHandles = ImmutableList.copyOf(metadata.getColumnHandles(session, tableHandle).values()); Map<String, Integer> columnIndex = indexColumns(columnHandles); assertTableIsBucketed(tableHandle); ImmutableMap<ColumnHandle, NullableValue> bindings = ImmutableMap.<ColumnHandle, NullableValue>builder() .put(columnHandles.get(columnIndex.get("t_float")), NullableValue.of(REAL, (long) floatToRawIntBits(87.1f))) .put(columnHandles.get(columnIndex.get("t_double")), NullableValue.of(DOUBLE, 88.2)) .build(); // floats and doubles are not supported, so we should see all splits MaterializedResult result = readTable(transaction, tableHandle, columnHandles, session, TupleDomain.fromFixedValues(bindings), OptionalInt.of(32), Optional.empty()); assertEquals(result.getRowCount(), 100); } }
private Optional<NullableValue> coerce(NullableValue value, Type targetType) { if (!TypeRegistry.canCoerce(value.getType(), targetType)) { return Optional.empty(); } if (value.isNull()) { return Optional.of(NullableValue.asNull(targetType)); } Object coercedValue = new FunctionInvoker(metadata.getFunctionRegistry()) .invoke(metadata.getFunctionRegistry().getCoercion(value.getType(), targetType), session.toConnectorSession(), value.getValue()); return Optional.of(NullableValue.of(targetType, coercedValue)); }
@Test public void testCreateMixedPredicate() { ImmutableList.Builder<HivePartition> partitions = ImmutableList.builder(); for (int i = 0; i < 5; i++) { partitions.add(new HivePartition( new SchemaTableName("test", "test"), Integer.toString(i), ImmutableMap.of(TEST_COLUMN_HANDLE, NullableValue.of(VarcharType.VARCHAR, Slices.utf8Slice(Integer.toString(i)))))); } partitions.add(new HivePartition( new SchemaTableName("test", "test"), "null", ImmutableMap.of(TEST_COLUMN_HANDLE, NullableValue.asNull(VarcharType.VARCHAR)))); createPredicate(ImmutableList.of(TEST_COLUMN_HANDLE), partitions.build()); } }