Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate files even if input is empty #6

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

imai-s
Copy link

@imai-s imai-s commented Feb 7, 2023

ローカルDBからレコードを取り出してGCSにアップロードするために、本プラグインを使用しようと考えています。
output は以下のように設定しています。

out:
  type: gcs
  bucket: {{ bucket }}
  path_prefix: {{ path_prefix }}
  file_ext: .jsonl
  formatter:
    type: jsonl
    charset: UTF-8

これは正常に動作しますが、取得したレコードがない場合(0個)うまく動作しません。

org.embulk.exec.PartialExecutionException: java.lang.NullPointerException
        at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(BulkLoader.java:340)
        at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:566)
        at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:35)
        at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:353)
        at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:350)
        at org.embulk.spi.Exec.doWith(Exec.java:23)
        at org.embulk.exec.BulkLoader.run(BulkLoader.java:350)
        at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:242)
        at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:291)
        at org.embulk.EmbulkRunner.run(EmbulkRunner.java:155)
        at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:431)
        at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:90)
        at org.embulk.cli.Main.main(Main.java:64)
Caused by: java.lang.NullPointerException
        at com.google.cloud.storage.StorageImpl.get(com/google/cloud/storage/StorageImpl.java:185)
        at com.google.cloud.storage.StorageImpl.get(com/google/cloud/storage/StorageImpl.java:202)
        at org.embulk.output.gcs.GcsTransactionalFileOutput.finish(org/embulk/output/gcs/GcsTransactionalFileOutput.java:116)
        at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:438)
        at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:302)
        at RUBY.finish(uri:classloader:/gems/embulk-0.9.24-java/lib/embulk/file_output.rb:44)
        at RUBY.finish(/root/.embulk/lib/gems/gems/embulk-formatter-jsonl-0.1.4/lib/embulk/formatter/jsonl.rb:77)
        at RUBY.finish(uri:classloader:/gems/embulk-0.9.24-java/lib/embulk/formatter_plugin.rb:80)
        at Embulk$$FormatterPlugin$$JavaAdapter$$OutputAdapter_1305971291.finish(Embulk$$FormatterPlugin$$JavaAdapter$$OutputAdapter_1305971291.gen:13)
        at org.embulk.spi.FileOutputRunner$DelegateTransactionalPageOutput.finish(org/embulk/spi/FileOutputRunner.java:154)
        at org.embulk.spi.PageBuilder.finish(org/embulk/spi/PageBuilder.java:236)
        at org.embulk.input.jdbc.AbstractJdbcInputPlugin.run(org/embulk/input/jdbc/AbstractJdbcInputPlugin.java:519)
        at org.embulk.spi.util.Executors.process(org/embulk/spi/util/Executors.java:62)
        at org.embulk.spi.util.Executors.process(org/embulk/spi/util/Executors.java:38)
        at org.embulk.exec.LocalExecutorPlugin$DirectExecutor$1.call(org/embulk/exec/LocalExecutorPlugin.java:170)
        at org.embulk.exec.LocalExecutorPlugin$DirectExecutor$1.call(org/embulk/exec/LocalExecutorPlugin.java:167)
        at java.util.concurrent.FutureTask.run(java/util/concurrent/FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java/util/concurrent/ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java/util/concurrent/ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(java/lang/Thread.java:750)

Error: java.lang.NullPointerException

Gemfile

source 'https://rubygems.org/'

gem 'embulk', '< 0.10'

gem 'embulk-input-postgresql', '~> 0.13.2'
gem 'embulk-output-gcs', '~> 0.6.0'
gem 'embulk-filter-to_json', '~> 0.1.0'
gem 'embulk-filter-ruby_proc', '~> 0.8.1'
gem 'embulk-formatter-jsonl', '~> 0.1.4'

原因は、初期化の際に @current_file == nil を設定していて、レコードがない場合変更されないからだと思っています。
そのため、GCSにアップロードするデータが nil のため Error: java.lang.NullPointerException が発生していそうです。

仕様に沿わない場合は、Closeしてください。

@imai-s imai-s closed this Feb 7, 2023
@imai-s imai-s reopened this Feb 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant