absoluteUri.getQuery(), absoluteUri.getFragment()); for (GcsPath entry : gcsUtil.expand(GcsPath.fromUri(gcsUriGlob))) { uris.add(entry.toUri());
absoluteUri.getQuery(), absoluteUri.getFragment()); for (GcsPath entry : gcsUtil.expand(GcsPath.fromUri(gcsUriGlob))) { uris.add(entry.toUri());
private GcsUtil buildMockGcsUtil() throws IOException { GcsUtil mockGcsUtil = mock(GcsUtil.class); when(mockGcsUtil.create(any(GcsPath.class), anyString())) .then( invocation -> FileChannel.open( Files.createTempFile("channel-", ".tmp"), StandardOpenOption.CREATE, StandardOpenOption.DELETE_ON_CLOSE)); when(mockGcsUtil.expand(any(GcsPath.class))) .then(invocation -> ImmutableList.of((GcsPath) invocation.getArguments()[0])); return mockGcsUtil; }
private GcsUtil buildMockGcsUtil() throws IOException { GcsUtil mockGcsUtil = Mockito.mock(GcsUtil.class); // Any request to open gets a new bogus channel Mockito.when(mockGcsUtil.open(Mockito.any(GcsPath.class))) .then( invocation -> FileChannel.open( Files.createTempFile("channel-", ".tmp"), StandardOpenOption.CREATE, StandardOpenOption.DELETE_ON_CLOSE)); // Any request for expansion returns a list containing the original GcsPath // This is required to pass validation that occurs in TextIO during apply() Mockito.when(mockGcsUtil.expand(Mockito.any(GcsPath.class))) .then(invocation -> ImmutableList.of((GcsPath) invocation.getArguments()[0])); return mockGcsUtil; } }
Mockito.when(mockGcsUtil.expand(Mockito.any(GcsPath.class))) .then( new Answer<List<GcsPath>>() {
absoluteUri.getQuery(), absoluteUri.getFragment()); for (GcsPath entry : gcsUtil.expand(GcsPath.fromUri(gcsUriGlob))) {
absoluteUri.getQuery(), absoluteUri.getFragment()); for (GcsPath entry : gcsUtil.expand(GcsPath.fromUri(gcsUriGlob))) {
/** * Delete multi file test output. * * @param outputPrefix prefix for files to delete * @throws Exception */ public void deleteOutputs(String outputPrefix) throws Exception { for (GcsPath path : gcsUtil.expand(GcsPath.fromUri(outputPrefix + "*"))) { deleteOutput(path.toString()); } }
@Test public void testNonExistentObjectReturnsEmptyResult() throws IOException { GcsOptions pipelineOptions = gcsOptionsWithTestCredential(); GcsUtil gcsUtil = pipelineOptions.getGcsUtil(); Storage mockStorage = Mockito.mock(Storage.class); gcsUtil.setStorageClient(mockStorage); Storage.Objects mockStorageObjects = Mockito.mock(Storage.Objects.class); Storage.Objects.Get mockStorageGet = Mockito.mock(Storage.Objects.Get.class); GcsPath pattern = GcsPath.fromUri("gs://testbucket/testdirectory/nonexistentfile"); GoogleJsonResponseException expectedException = googleJsonResponseException( HttpStatusCodes.STATUS_CODE_NOT_FOUND, "It don't exist", "Nothing here to see"); when(mockStorage.objects()).thenReturn(mockStorageObjects); when(mockStorageObjects.get(pattern.getBucket(), pattern.getObject())) .thenReturn(mockStorageGet); when(mockStorageGet.execute()).thenThrow(expectedException); assertEquals(Collections.emptyList(), gcsUtil.expand(pattern)); }
StandardOpenOption.DELETE_ON_CLOSE)); when(mockGcsUtil.expand(any(GcsPath.class))) .then(invocation -> ImmutableList.of((GcsPath) invocation.getArguments()[0])); when(mockGcsUtil.bucketAccessible(GcsPath.fromUri(VALID_STAGING_BUCKET))).thenReturn(true);
@Test public void testAccessDeniedObjectThrowsIOException() throws IOException { GcsOptions pipelineOptions = gcsOptionsWithTestCredential(); GcsUtil gcsUtil = pipelineOptions.getGcsUtil(); Storage mockStorage = Mockito.mock(Storage.class); gcsUtil.setStorageClient(mockStorage); Storage.Objects mockStorageObjects = Mockito.mock(Storage.Objects.class); Storage.Objects.Get mockStorageGet = Mockito.mock(Storage.Objects.Get.class); GcsPath pattern = GcsPath.fromUri("gs://testbucket/testdirectory/accessdeniedfile"); GoogleJsonResponseException expectedException = googleJsonResponseException( HttpStatusCodes.STATUS_CODE_FORBIDDEN, "Waves hand mysteriously", "These aren't the buckets you're looking for"); when(mockStorage.objects()).thenReturn(mockStorageObjects); when(mockStorageObjects.get(pattern.getBucket(), pattern.getObject())) .thenReturn(mockStorageGet); when(mockStorageGet.execute()).thenThrow(expectedException); thrown.expect(IOException.class); thrown.expectMessage("Unable to get the file object for path"); gcsUtil.expand(pattern); }
private static DataflowPipelineOptions buildPipelineOptions(String... args) throws IOException { GcsUtil mockGcsUtil = mock(GcsUtil.class); when(mockGcsUtil.expand(any(GcsPath.class))) .then(invocation -> ImmutableList.of((GcsPath) invocation.getArguments()[0])); when(mockGcsUtil.bucketAccessible(any(GcsPath.class))).thenReturn(true); DataflowPipelineOptions options = PipelineOptionsFactory.fromArgs(args).as(DataflowPipelineOptions.class); options.setRunner(DataflowRunner.class); options.setGcpCredential(new TestCredential()); options.setJobName("some-job-name"); options.setProject("some-project"); options.setRegion("some-region"); options.setTempLocation(GcsPath.fromComponents("somebucket", "some/path").toString()); options.setFilesToStage(new ArrayList<>()); options.setGcsUtil(mockGcsUtil); return options; } }
private static DataflowPipelineOptions buildPipelineOptions() throws IOException { GcsUtil mockGcsUtil = mock(GcsUtil.class); when(mockGcsUtil.expand(any(GcsPath.class))) .then(invocation -> ImmutableList.of((GcsPath) invocation.getArguments()[0])); when(mockGcsUtil.bucketAccessible(any(GcsPath.class))).thenReturn(true); DataflowPipelineOptions options = PipelineOptionsFactory.as(DataflowPipelineOptions.class); options.setRunner(DataflowRunner.class); options.setGcpCredential(new TestCredential()); options.setJobName("some-job-name"); options.setProject("some-project"); options.setRegion("some-region"); options.setTempLocation(GcsPath.fromComponents("somebucket", "some/path").toString()); options.setFilesToStage(new ArrayList<>()); options.setDataflowClient(buildMockDataflow(new IsValidCreateRequest())); options.setGcsUtil(mockGcsUtil); // Enable the FileSystems API to know about gs:// URIs in this test. FileSystems.setDefaultPipelineOptions(options); return options; }
try { results = Lists.newArrayList(); for (GcsPath path : gcsUtil.expand(GcsPath.fromUri(outputPrefix + "*"))) { BufferedReader reader = openOutput(path.toString()); for (String line = reader.readLine(); line != null; line = reader.readLine()) {
ImmutableList.of(GcsPath.fromUri("gs://testbucket/testdirectory/otherfile")); assertThat(expectedFiles, contains(gcsUtil.expand(pattern).toArray())); GcsPath.fromUri("gs://testbucket/testdirectory/file3name")); assertThat(expectedFiles, contains(gcsUtil.expand(pattern).toArray())); GcsPath.fromUri("gs://testbucket/testdirectory/file3name")); assertThat(expectedFiles, contains(gcsUtil.expand(pattern).toArray())); GcsPath.fromUri("gs://testbucket/testdirectory/file3name")); assertThat(expectedFiles, contains(gcsUtil.expand(pattern).toArray())); GcsPath.fromUri("gs://testbucket/testdirectory/file3name")); assertThat(expectedFiles, contains(gcsUtil.expand(pattern).toArray()));
GcsPath.fromUri("gs://testbucket/test/file4.txt")); assertThat(expectedFiles, contains(gcsUtil.expand(pattern).toArray()));