/** * create ext table snapshot * @param signature * @param tableName * @param keyColumns *@param storageType * @param storageLocation @return created snapshot * @throws IOException */ public ExtTableSnapshotInfo createSnapshot(TableSignature signature, String tableName, String snapshotID, String[] keyColumns, int shardNum, String storageType, String storageLocation) throws IOException { ExtTableSnapshotInfo snapshot = new ExtTableSnapshotInfo(); snapshot.setUuid(snapshotID); snapshot.setSignature(signature); snapshot.setTableName(tableName); snapshot.setKeyColumns(keyColumns); snapshot.setStorageType(storageType); snapshot.setStorageLocationIdentifier(storageLocation); snapshot.setShardNum(shardNum); save(snapshot); return snapshot; }
private ExtTableSnapshotInfo buildSnapshotCache(String snapshotID, int rowCnt) throws Exception { ExtTableSnapshotInfo snapshotInfo = new ExtTableSnapshotInfo(); snapshotInfo.setTableName(TABLE_COUNTRY); snapshotInfo.setUuid(snapshotID); snapshotInfo.setStorageType(MOCK_EXT_LOOKUP); snapshotInfo.setKeyColumns(new String[] { "COUNTRY" }); snapshotInfo.setRowCnt(rowCnt); snapshotInfo.setSignature(new TableSignature("/test", rowCnt, System.currentTimeMillis())); ExtTableSnapshotInfoManager.getInstance(kylinConfig).save(snapshotInfo); RocksDBLookupTableCache cache = RocksDBLookupTableCache.getInstance(kylinConfig); cache.buildSnapshotCache(tableDesc, snapshotInfo, getLookupTableWithRandomData(rowCnt)); while (cache.getCacheState(snapshotInfo) == CacheState.IN_BUILDING) { Thread.sleep(500); } return snapshotInfo; }
@Test public void testRestoreCacheFromFiles() throws Exception { String snapshotID = RandomUtil.randomUUID().toString(); String snapshotCacheBasePath = RocksDBLookupTableCache.getCacheBasePath(kylinConfig) + File.separator + TABLE_COUNTRY + File.separator + snapshotID; String dbPath = snapshotCacheBasePath + File.separator + "db"; RocksDBLookupBuilder builder = new RocksDBLookupBuilder(tableDesc, new String[] { "COUNTRY" }, dbPath); builder.build(getLookupTableWithRandomData(10000)); String stateFilePath = snapshotCacheBasePath + File.separator + "STATE"; Files.write(CacheState.AVAILABLE.name(), new File(stateFilePath), Charsets.UTF_8); RocksDBLookupTableCache cache = RocksDBLookupTableCache.getInstance(kylinConfig); ExtTableSnapshotInfo snapshotInfo = new ExtTableSnapshotInfo(); snapshotInfo.setTableName(TABLE_COUNTRY); snapshotInfo.setUuid(snapshotID); snapshotInfo.setStorageType(MOCK_EXT_LOOKUP); snapshotInfo.setKeyColumns(new String[] { "COUNTRY" }); ILookupTable lookupTable = cache.getCachedLookupTable(tableDesc, snapshotInfo, false); int rowCnt = 0; for (String[] strings : lookupTable) { rowCnt++; } lookupTable.close(); assertEquals(10000, rowCnt); }
/** * create ext table snapshot * @param signature * @param tableName * @param keyColumns *@param storageType * @param storageLocation @return created snapshot * @throws IOException */ public ExtTableSnapshotInfo createSnapshot(TableSignature signature, String tableName, String snapshotID, String[] keyColumns, int shardNum, String storageType, String storageLocation) throws IOException { ExtTableSnapshotInfo snapshot = new ExtTableSnapshotInfo(); snapshot.setUuid(snapshotID); snapshot.setSignature(signature); snapshot.setTableName(tableName); snapshot.setKeyColumns(keyColumns); snapshot.setStorageType(storageType); snapshot.setStorageLocationIdentifier(storageLocation); snapshot.setShardNum(shardNum); save(snapshot); return snapshot; }