Key Default Description
io.manager.async.num-read-write-thread
-1 The number of async read write thread. If not positive, it will be adjusted to max(1, number of temp dirs) for TM shuffle and max(2, 2 * number of disks) for YARN shuffle.
io.manager.buffered.read.size
-1 The buffer size of io manager buffered read, -1 mean not use buffered read, this will reduce random IO, but will result in more than one copy.
io.manager.buffered.write.size
-1 The buffer size of io manager buffered write, -1 mean not use buffered write, this will reduce random IO, but will result in more than one copy.
task.blocking.shuffle.type
"TM" The type of shuffle service used for blocking edge. Currently it can be configured to TM or YARN.
task.cancellation.interval
30000 Time interval between two successive task cancellation attempts in milliseconds.
task.cancellation.timeout
180000 Timeout in milliseconds after which a task cancellation times out and leads to a fatal TaskManager error. A value of 0 deactivates the watch dog.
task.cancellation.timers.timeout
7500
task.checkpoint.alignment.max-size
-1 The maximum number of bytes that a checkpoint alignment may buffer. If the checkpoint alignment buffers more than the configured amount of data, the checkpoint is aborted (skipped). A value of -1 indicates that there is no limit.
task.external.shuffle.compression.buffer-size
65536 The max buffer size to compress external shuffle data.
task.external.shuffle.compression.codec
"lz4" The codec to use when compress or decompress external shuffle data. Currently supported codecs are lz4, bzip2, gzip. User can also implement interface BlockCompressionFactory and set its class to specify other codecs.
task.external.shuffle.compression.enable
true Whether to enable compress shuffle data when using external shuffle.
task.external.shuffle.consumed-partition-ttl-in-seconds
3600 The time interval to delete the fully consumed shuffle data directories since they become inactive.
task.external.shuffle.max-concurrent-requests
2000 The maximum number of concurrent requests in the reduce-side tasks.
task.external.shuffle.partial-consumed-partition-ttl-in-seconds
43200 The time interval to delete the partially consumed shuffle data directories since they become inactive.
task.external.shuffle.unconsumed-partition-ttl-in-seconds
43200 TThe time interval to delete the unconsumed shuffle data directories since they are ready to consume.
task.external.shuffle.unfinished-partition-ttl-in-seconds
3600 The time interval to delete the writing shuffle data directories since the last writing.
taskmanager.capacity.cpu.core
-1.0 The overall cpu cores allocated to the task manager.
taskmanager.capacity.memory.mb
-1 The overall memory in MB that allocated to the task manager.
taskmanager.cpu.core
1.0 How many physical cpu cores a task manager will supply for user
taskmanager.data.port
0 The task manager’s port used for data exchange operations.
taskmanager.data.ssl.enabled
true Enable SSL support for the taskmanager data transport. This is applicable only when the global ssl flag security.ssl.enabled is set to true
taskmanager.debug.memory.log
false Flag indicating whether to start a thread, which repeatedly logs the memory usage of the JVM.
taskmanager.debug.memory.log-interval
5000 The interval (in ms) for the log thread to log the current memory usage.
taskmanager.direct.memory.mb
0 How many direct memory (in megabytes) a task manager will supply for user.
taskmanager.exit-on-fatal-akka-error
false Whether the quarantine monitor for task managers shall be started. The quarantine monitor shuts down the actor system if it detects that it has quarantined another actor system or if it has been quarantined by another actor system.
taskmanager.extended.resources
(none) Extended resources will supply for user. Specified as resource-type:value pairs separated by commas. such as GPU:1,FPGA:1.
taskmanager.floating.memory.size
0
taskmanager.heap.mb
1024 How many heap memory (in megabytes) a task manager will supply for user, not including managed memory.
taskmanager.host
(none) The hostname of the network interface that the TaskManager binds to. By default, the TaskManager searches for network interfaces that can connect to the JobManager and other TaskManagers. This option can be used to define a hostname if that strategy fails for some reason. Because different TaskManagers need different values for this option, it usually is specified in an additional non-shared TaskManager-specific config file.
taskmanager.jvm-exit-on-oom
false Whether to kill the TaskManager when the task thread throws an OutOfMemoryError.
taskmanager.jvm.memory.dynamic.young.ratio
0.25 Ratio of young generation for dynamic memory in task manager.
taskmanager.jvm.memory.persistent.young.ratio
0.1 Ratio of young generation for persistent memory in task manager.
taskmanager.managed.memory.size
-1 Amount of memory to be allocated by the task manager's memory manager (in megabytes). If not set, a relative fraction will be allocated.
taskmanager.memory.fraction
0.7 The relative amount of memory (after subtracting the amount of memory used by network buffers) that the task manager reserves for sorting, hash tables, and caching of intermediate results. For example, a value of `0.8` means that a task manager reserves 80% of its memory for internal data buffers, leaving 20% of free memory for the task manager's heap for objects created by user-defined functions. This parameter is only evaluated, if taskmanager.managed.memory.size is not set.
taskmanager.memory.off-heap
false Memory allocation method (JVM heap or off-heap), used for managed memory of the TaskManager as well as the network buffers.
taskmanager.memory.preallocate
false Whether TaskManager managed memory should be pre-allocated when the TaskManager is starting.
taskmanager.memory.segment-size
32768 Size of memory buffers used by the network stack and the memory manager (in bytes).
taskmanager.multi-slots.max.cpu.core
1.0 Cpu core limitation, used to decide how many slots can be placed on a taskmanager.
taskmanager.multi-slots.max.extended-resources
(none) Extended resources limitation, used to decide how many slots can be placed on a taskmanger. String format is like "GPU=10,FPGA=12".
taskmanager.multi-slots.max.memory.mb
32768 Memory (in megabytes) limitation, used to decide how many slots can be placed on a taskmanager.
taskmanager.multi-slots.min.cpu.core
1.0 Min cpu core for a taskmanager.
taskmanager.multi-slots.min.memory.mb
1024 Min memory (in megabytes) for taskmanager.
taskmanager.native.memory.mb
0 How many native memory (in megabytes) a task manager will supply for user.
taskmanager.network.check-partition-producer-state
false Boolean flag indicates whether to check partition producer state if the task requests a partition failed and wants to re-trigger the partition request. The task will re-trigger the partition request if the producer is healthy or fail otherwise.
taskmanager.network.detailed-metrics
false Boolean flag to enable/disable more detailed metrics about inbound/outbound network queue lengths.
taskmanager.network.memory.buffers-per-channel
2 Maximum number of network buffers to use for each outgoing/incoming channel (subpartition/input channel).In credit-based flow control mode, this indicates how many credits are exclusive in each input channel. It should be configured at least 2 for good performance. 1 buffer is for receiving in-flight data in the subpartition and 1 buffer is for parallel serialization.
taskmanager.network.memory.buffers-per-external-blocking-channel
16 The number of buffers available for each external blocking channel.
taskmanager.network.memory.buffers-per-subpartition
2
taskmanager.network.memory.floating-buffers-per-external-blocking-gate
0 taskmanager.network.memory.floating-buffers-per-external-blocking-gate
taskmanager.network.memory.floating-buffers-per-gate
8 Number of extra network buffers to use for each outgoing/incoming gate (result partition/input gate). In credit-based flow control mode, this indicates how many floating credits are shared among all the input channels. The floating buffers are distributed based on backlog (real-time output buffers in the subpartition) feedback, and can help relieve back-pressure caused by unbalanced data distribution among the subpartitions. This value should be increased in case of higher round trip times between nodes and/or larger number of machines in the cluster.
taskmanager.network.memory.fraction
0.1 Fraction of JVM memory to use for network buffers. This determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value or the min/max values below. Also note, that "taskmanager.network.memory.min"` and "taskmanager.network.memory.max" may override this fraction.
taskmanager.network.memory.max
1073741824 Maximum memory size for network buffers (in bytes).
taskmanager.network.memory.min
67108864 Minimum memory size for network buffers (in bytes).
taskmanager.network.request-backoff.initial
100 Minimum backoff for partition requests of input channels.
taskmanager.network.request-backoff.max
10000 Maximum backoff for partition requests of input channels.
taskmanager.numberOfTaskSlots
1 The number of parallel operator or user function instances that a single TaskManager can run. If this value is larger than 1, a single TaskManager takes multiple instances of a function or operator. That way, the TaskManager can utilize multiple CPU cores, but at the same time, the available memory is divided between the different operator or function instances. This value is typically proportional to the number of physical CPU cores that the TaskManager's machine has (e.g., equal to the number of cores, or half the number of cores).
taskmanager.output.hash.max-subpartitions
200 The maximum number of subpartitions supported by the hash writer.
taskmanager.output.local-disk.type
(none) The disk type preferred to write the shuffle data. If not specified, all the root directories are feasible. If specified, only directories with the configured type are feasible.
taskmanager.output.local-output-dirs
(none) The available directories for the external shuffle service. It will be configured automatically and should not be configured manually.
taskmanager.output.memory.mb
200 The write buffer size for each output in a task.
taskmanager.output.merge.enable-async-merge
false Whether to start merge while writing has not been finished.
taskmanager.output.merge.factor
64 The maximum number of files to merge at once when using the merge writer.
taskmanager.output.merge.merge-to-one-file
true Whether to merge to one file finally when using the merge writer. If not, the merge stops once the number of files are less than taskmanager.output.merge.factor.
taskmanager.process.heap.memory.mb
128 The heap memory (in megabytes) used for task manager process.
taskmanager.process.native.memory.mb
0 The native memory (in megabytes) used for task manager process.
taskmanager.process.netty.memory.mb
64 The direct memory (in megabytes) used for netty framework in the task manager process.
taskmanager.reconnection.timeout
"1 min" Defines the maximum time it can take for the TaskManager reconnection. If the duration is exceeded without a successful reconnection, then disassociate from JM.
taskmanager.registration.initial-backoff
"500 ms" The initial registration backoff between two consecutive registration attempts. The backoff is doubled for each new registration attempt until it reaches the maximum registration backoff.
taskmanager.registration.max-backoff
"30 s" The maximum registration backoff between two consecutive registration attempts. The max registration backoff requires a time unit specifier (ms/s/min/h/d).
taskmanager.registration.refused-backoff
"10 s" The backoff after a registration has been refused by the job manager before retrying to connect.
taskmanager.registration.timeout
"5 min" Defines the timeout for the TaskManager registration. If the duration is exceeded without a successful registration, then the TaskManager terminates.
taskmanager.resourceProfile
(none) The resource profile of a slot in a task executor.
taskmanager.rpc.port
"0" The task manager’s IPC port. Accepts a list of ports (“50100,50101”), ranges (“50100-50200”) or a combination of both. It is recommended to set a range of ports to avoid collisions when multiple TaskManagers are running on the same machine.
taskmanager.total.resourceProfile
(none) The total resource profile of all the slots in a task executor.