Package org.apache.spark.sql.streaming

Class StreamingQueryProgress

Object
org.apache.spark.sql.streaming.StreamingQueryProgress
All Implemented Interfaces:
Serializable
public class StreamingQueryProgress extends Object implements Serializable
Information about progress made in the execution of a StreamingQuery during a trigger. Each event relates to processing done for a single trigger of the streaming query. Events are emitted even when no new data is available to be processed.

param: id A unique query id that persists across restarts. See StreamingQuery.id(). param: runId A query id that is unique for every start/restart. See StreamingQuery.runId(). param: name User-specified name of the query, null if not specified. param: timestamp Beginning time of the trigger in ISO8601 format, i.e. UTC timestamps. param: batchId A unique id for the current batch of data being processed. Note that in the case of retries after a failure a given batchId my be executed more than once. Similarly, when there is no data to be processed, the batchId will not be incremented. param: batchDuration The process duration of each batch. param: durationMs The amount of time taken to perform various operations in milliseconds. param: eventTime Statistics of event time seen in this batch. It may contain the following keys: 


                   "max" -> "2016-12-05T20:54:20.827Z"  // maximum event time seen in this trigger
                   "min" -> "2016-12-05T20:54:20.827Z"  // minimum event time seen in this trigger
                   "avg" -> "2016-12-05T20:54:20.827Z"  // average event time seen in this trigger
                   "watermark" -> "2016-12-05T20:54:20.827Z"  // watermark used in this trigger
All timestamps are in ISO8601 format, i.e. UTC timestamps. param: stateOperators Information about operators in the query that store state. param: sources detailed statistics on data being read from each of the streaming sources.
Since:
2.1.0
See Also:

  • Method Details

    • id

      public UUID id()

    • runId

      public UUID runId()

    • name

      public String name()

    • timestamp

      public String timestamp()

    • batchId

      public long batchId()

    • batchDuration

      public long batchDuration()

    • durationMs

      public Map<String,Long> durationMs()

    • eventTime

      public Map<String,String> eventTime()

    • stateOperators

      public StateOperatorProgress[] stateOperators()

    • sources

      public SourceProgress[] sources()

    • sink

      public SinkProgress sink()

    • observedMetrics

      public Map<String,Row> observedMetrics()

    • numInputRows

      public long numInputRows()
      The aggregate (across all sources) number of records processed in a trigger.

    • inputRowsPerSecond

      public double inputRowsPerSecond()
      The aggregate (across all sources) rate of data arriving.

    • processedRowsPerSecond

      public double processedRowsPerSecond()
      The aggregate (across all sources) rate at which Spark is processing data.

    • json

      public String json()
      The compact JSON representation of this progress.

    • prettyJson

      public String prettyJson()
      The pretty (i.e. indented) JSON representation of this progress.

    • toString

      public String toString()
      Overrides:
      toString in class Object