当前位置:   article > 正文

libkafka的配置项_kafkareader error to set receive.message.max.bytes

kafkareader error to set receive.message.max.bytes configuration property re

libkafka的配置项

//@file

Global configuration properties

PropertyC/PRangeDefaultDescription
builtin.features*gzip, snappy, ssl, sasl, regex, lz4, sasl_gssapi, sasl_plain, sasl_scram, pluginsIndicates the builtin features for this build of librdkafka. An application can either query this value or attempt to set it with its list of required features to check for library support.
Type: CSV flags
client.id*rdkafkaClient identifier.
Type: string
metadata.broker.list*Initial list of brokers as a CSV list of broker host or host:port. The application may also use rd_kafka_brokers_add() to add brokers during runtime.
Type: string
bootstrap.servers*Alias for metadata.broker.list
message.max.bytes*1000 .. 10000000001000000Maximum Kafka protocol request message size.
Type: integer
message.copy.max.bytes*0 .. 100000000065535Maximum size for message to be copied to buffer. Messages larger than this will be passed by reference (zero-copy) at the expense of larger iovecs.
Type: integer
receive.message.max.bytes*1000 .. 2147483647100000000Maximum Kafka protocol response message size. This serves as a safety precaution to avoid memory exhaustion in case of protocol hickups. This value is automatically adjusted upwards to be at least fetch.max.bytes + 512 to allow for protocol overhead.
Type: integer
max.in.flight.requests.per.connection*1 .. 10000001000000Maximum number of in-flight requests per broker connection. This is a generic property applied to all broker communication, however it is primarily relevant to produce requests. In particular, note that other mechanisms limit the number of outstanding consumer fetch request per broker to one.
Type: integer
max.in.flight*Alias for max.in.flight.requests.per.connection
metadata.request.timeout.ms*10 .. 90000060000Non-topic request timeout in milliseconds. This is for metadata requests, etc.
Type: integer
topic.metadata.refresh.interval.ms*-1 .. 3600000300000Topic metadata refresh interval in milliseconds. The metadata is automatically refreshed on error and connect. Use -1 to disable the intervalled refresh.
Type: integer
metadata.max.age.ms*1 .. 86400000-1Metadata cache max age. Defaults to topic.metadata.refresh.interval.ms * 3
Type: integer
topic.metadata.refresh.fast.interval.ms*1 .. 60000250When a topic loses its leader a new metadata request will be enqueued with this initial interval, exponentially increasing until the topic metadata has been refreshed. This is used to recover quickly from transitioning leader brokers.
Type: integer
topic.metadata.refresh.fast.cnt*0 .. 100010Deprecated: No longer used.
Type: integer
topic.metadata.refresh.sparse*true, falsetrueSparse metadata requests (consumes less network bandwidth)
Type: boolean
topic.blacklist*Topic blacklist, a comma-separated list of regular expressions for matching topic names that should be ignored in broker metadata information as if the topics did not exist.
Type: pattern list
debug*generic, broker, topic, metadata, feature, queue, msg, protocol, cgrp, security, fetch, interceptor, plugin, consumer, admin, allA comma-separated list of debug contexts to enable. Detailed Producer debugging: broker,topic,msg. Consumer: consumer,cgrp,topic,fetch
Type: CSV flags
socket.timeout.ms*10 .. 30000060000Default timeout for network requests. Producer: ProduceRequests will use the lesser value of socket.timeout.ms and remaining message.timeout.ms for the first message in the batch. Consumer: FetchRequests will use fetch.wait.max.ms + socket.timeout.ms. Admin: Admin requests will use socket.timeout.ms or explicitly set rd_kafka_AdminOptions_set_operation_timeout() value.
Type: integer
socket.blocking.max.ms*1 .. 600001000Maximum time a broker socket operation may block. A lower value improves responsiveness at the expense of slightly higher CPU usage. Deprecated
Type: integer
socket.send.buffer.bytes*0 .. 1000000000Broker socket send buffer size. System default is used if 0.
Type: integer
socket.receive.buffer.bytes*0 .. 1000000000Broker socket receive buffer size. System default is used if 0.
Type: integer
socket.keepalive.enable*true, falsefalseEnable TCP keep-alives (SO_KEEPALIVE) on broker sockets
Type: boolean
socket.nagle.disable*true, falsefalseDisable the Nagle algorithm (TCP_NODELAY) on broker sockets.
Type: boolean
socket.max.fails*0 .. 10000001Disconnect from broker when this number of send failures (e.g., timed out requests) is reached. Disable with 0. NOTE: The connection is automatically re-established.
Type: integer
broker.address.ttl*0 .. 864000001000How long to cache the broker address resolving results (milliseconds).
Type: integer
broker.address.family*any, v4, v6anyAllowed broker IP address families: any, v4, v6
Type: enum value
reconnect.backoff.jitter.ms*0 .. 3600000500Throttle broker reconnection attempts by this value +-50%.
Type: integer
statistics.interval.ms*0 .. 864000000librdkafka statistics emit interval. The application also needs to register a stats callback using rd_kafka_conf_set_stats_cb(). The granularity is 1000ms. A value of 0 disables statistics.
Type: integer
enabled_events*0 .. 21474836470See rd_kafka_conf_set_events()
Type: integer
error_cb*Error callback (set with rd_kafka_conf_set_error_cb())
Type: pointer
throttle_cb*Throttle callback (set with rd_kafka_conf_set_throttle_cb())
Type: pointer
stats_cb*Statistics callback (set with rd_kafka_conf_set_stats_cb())
Type: pointer
log_cb*Log callback (set with rd_kafka_conf_set_log_cb())
Type: pointer
log_level*0 .. 76Logging level (syslog(3) levels)
Type: integer
log.queue*true, falsefalseDisable spontaneous log_cb from internal librdkafka threads, instead enqueue log messages on queue set with rd_kafka_set_log_queue() and serve log callbacks or events through the standard poll APIs. NOTE: Log messages will linger in a temporary queue until the log queue has been set.
Type: boolean
log.thread.name*true, falsetruePrint internal thread name in log messages (useful for debugging librdkafka internals)
Type: boolean
log.connection.close*true, falsetrueLog broker disconnects. It might be useful to turn this off when interacting with 0.9 brokers with an aggressive connection.max.idle.ms value.
Type: boolean
background_event_cb*Background queue event callback (set with rd_kafka_conf_set_background_event_cb())
Type: pointer
socket_cb*Socket creation callback to provide race-free CLOEXEC
Type: pointer
connect_cb*Socket connect callback
Type: pointer
closesocket_cb*Socket close callback
Type: pointer
open_cb*File open callback to provide race-free CLOEXEC
Type: pointer
opaque*Application opaque (set with rd_kafka_conf_set_opaque())
Type: pointer
default_topic_conf*Default topic configuration for automatically subscribed topics
Type: pointer
internal.termination.signal*0 .. 1280Signal that librdkafka will use to quickly terminate on rd_kafka_destroy(). If this signal is not set then there will be a delay before rd_kafka_wait_destroyed() returns true as internal threads are timing out their system calls. If this signal is set however the delay will be minimal. The application should mask this signal as an internal signal handler is installed.
Type: integer
api.version.request*true, falsetrueRequest broker’s supported API versions to adjust functionality to available protocol features. If set to false, or the ApiVersionRequest fails, the fallback version broker.version.fallback will be used. NOTE: Depends on broker version >=0.10.0. If the request is not supported by (an older) broker the broker.version.fallback fallback is used.
Type: boolean
api.version.request.timeout.ms*1 .. 30000010000Timeout for broker API version requests.
Type: integer
api.version.fallback.ms*0 .. 6048000001200000Dictates how long the broker.version.fallback fallback is used in the case the ApiVersionRequest fails. NOTE: The ApiVersionRequest is only issued when a new connection to the broker is made (such as after an upgrade).
Type: integer
broker.version.fallback*0.9.0Older broker versions (<0.10.0) provides no way for a client to query for supported protocol features (ApiVersionRequest, see api.version.request) making it impossible for the client to know what features it may use. As a workaround a user may set this property to the expected broker version and the client will automatically adjust its feature set accordingly if the ApiVersionRequest fails (or is disabled). The fallback broker version will be used for api.version.fallback.ms. Valid values are: 0.9.0, 0.8.2, 0.8.1, 0.8.0. Any other value, such as 0.10.2.1, enables ApiVersionRequests.
Type: string
security.protocol*plaintext, ssl, sasl_plaintext, sasl_sslplaintextProtocol used to communicate with brokers.
Type: enum value
ssl.cipher.suites*A cipher suite is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol. See manual page for ciphers(1) and `SSL_CTX_set_cipher_list(3).
Type: string
ssl.curves.list*The supported-curves extension in the TLS ClientHello message specifies the curves (standard/named, or ‘explicit’ GF(2^k) or GF(p)) the client is willing to have the server use. See manual page for SSL_CTX_set1_curves_list(3). OpenSSL >= 1.0.2 required.
Type: string
ssl.sigalgs.list*The client uses the TLS ClientHello signature_algorithms extension to indicate to the server which signature/hash algorithm pairs may be used in digital signatures. See manual page for SSL_CTX_set1_sigalgs_list(3). OpenSSL >= 1.0.2 required.
Type: string
ssl.key.location*Path to client’s private key (PEM) used for authentication.
Type: string
ssl.key.password*Private key passphrase
Type: string
ssl.certificate.location*Path to client’s public key (PEM) used for authentication.
Type: string
ssl.ca.location*File or directory path to CA certificate(s) for verifying the broker’s key.
Type: string
ssl.crl.location*Path to CRL for verifying broker’s certificate validity.
Type: string
ssl.keystore.location*Path to client’s keystore (PKCS#12) used for authentication.
Type: string
ssl.keystore.password*Client’s keystore (PKCS#12) password.
Type: string
sasl.mechanisms*GSSAPISASL mechanism to use for authentication. Supported: GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512. NOTE: Despite the name only one mechanism must be configured.
Type: string
sasl.mechanism*Alias for sasl.mechanisms
sasl.kerberos.service.name*kafkaKerberos principal name that Kafka runs as, not including /hostname@REALM
Type: string
sasl.kerberos.principal*kafkaclientThis client’s Kerberos principal name. (Not supported on Windows, will use the logon user’s principal).
Type: string
sasl.kerberos.kinit.cmd*kinit -S “%{sasl.kerberos.service.name}/%{broker.name}” -k -t “%{sasl.kerberos.keytab}” %{sasl.kerberos.principal}Full kerberos kinit command string, %{config.prop.name} is replaced by corresponding config object value, %{broker.name} returns the broker’s hostname.
Type: string
sasl.kerberos.keytab*Path to Kerberos keytab file. Uses system default if not set.NOTE: This is not automatically used but must be added to the template in sasl.kerberos.kinit.cmd as ... -t %{sasl.kerberos.keytab}.
Type: string
sasl.kerberos.min.time.before.relogin*1 .. 8640000060000Minimum time in milliseconds between key refresh attempts.
Type: integer
sasl.username*SASL username for use with the PLAIN and SASL-SCRAM-.. mechanisms
Type: string
sasl.password*SASL password for use with the PLAIN and SASL-SCRAM-.. mechanism
Type: string
plugin.library.paths*List of plugin libaries to load (; separated). The library search path is platform dependent (see dlopen(3) for Unix and LoadLibrary() for Windows). If no filename extension is specified the platform-specific extension (such as .dll or .so) will be appended automatically.
Type: string
interceptors*Interceptors added through rd_kafka_conf_interceptor_add_..() and any configuration handled by interceptors.
Type:
group.id*Client group id string. All clients sharing the same group.id belong to the same group.
Type: string
partition.assignment.strategy*range,roundrobinName of partition assignment strategy to use when elected group leader assigns partitions to group members.
Type: string
session.timeout.ms*1 .. 360000030000Client group session and failure detection timeout.
Type: integer
heartbeat.interval.ms*1 .. 36000001000Group session keepalive heartbeat interval.
Type: integer
group.protocol.type*consumerGroup protocol type
Type: string
coordinator.query.interval.ms*1 .. 3600000600000How often to query for the current client group coordinator. If the currently assigned coordinator is down the configured query interval will be divided by ten to more quickly recover in case of coordinator reassignment.
Type: integer
enable.auto.commitCtrue, falsetrueAutomatically and periodically commit offsets in the background. Note: setting this to false does not prevent the consumer from fetching previously committed start offsets. To circumvent this behaviour set specific start offsets per partition in the call to assign().
Type: boolean
auto.commit.interval.msC0 .. 864000005000The frequency in milliseconds that the consumer offsets are committed (written) to offset storage. (0 = disable). This setting is used by the high-level consumer.
Type: integer
enable.auto.offset.storeCtrue, falsetrueAutomatically store offset of last message provided to application.
Type: boolean
queued.min.messagesC1 .. 10000000100000Minimum number of messages per topic+partition librdkafka tries to maintain in the local consumer queue.
Type: integer
queued.max.messages.kbytesC1 .. 20971511048576Maximum number of kilobytes per topic+partition in the local consumer queue. This value may be overshot by fetch.message.max.bytes. This property has higher priority than queued.min.messages.
Type: integer
fetch.wait.max.msC0 .. 300000100Maximum time the broker may wait to fill the response with fetch.min.bytes.
Type: integer
fetch.message.max.bytesC1 .. 10000000001048576Initial maximum number of bytes per topic+partition to request when fetching messages from the broker. If the client encounters a message larger than this value it will gradually try to increase it until the entire message can be fetched.
Type: integer
max.partition.fetch.bytesCAlias for fetch.message.max.bytes
fetch.max.bytesC0 .. 214748313552428800Maximum amount of data the broker shall return for a Fetch request. Messages are fetched in batches by the consumer and if the first message batch in the first non-empty partition of the Fetch request is larger than this value, then the message batch will still be returned to ensure the consumer can make progress. The maximum message batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (broker topic config). fetch.max.bytes is automatically adjusted upwards to be at least message.max.bytes (consumer config).
Type: integer
fetch.min.bytesC1 .. 1000000001Minimum number of bytes the broker responds with. If fetch.wait.max.ms expires the accumulated data will be sent to the client regardless of this setting.
Type: integer
fetch.error.backoff.msC0 .. 300000500How long to postpone the next fetch request for a topic+partition in case of a fetch error.
Type: integer
offset.store.methodCnone, file, brokerbrokerOffset commit store method: ‘file’ - local file store (offset.store.path, et.al), ‘broker’ - broker commit store (requires Apache Kafka 0.8.2 or later on the broker).
Type: enum value
consume_cbCMessage consume callback (set with rd_kafka_conf_set_consume_cb())
Type: pointer
rebalance_cbCCalled after consumer group has been rebalanced (set with rd_kafka_conf_set_rebalance_cb())
Type: pointer
offset_commit_cbCOffset commit result propagation callback. (set with rd_kafka_conf_set_offset_commit_cb())
Type: pointer
enable.partition.eofCtrue, falsetrueEmit RD_KAFKA_RESP_ERR__PARTITION_EOF event whenever the consumer reaches the end of a partition.
Type: boolean
check.crcsCtrue, falsefalseVerify CRC32 of consumed messages, ensuring no on-the-wire or on-disk corruption to the messages occurred. This check comes at slightly increased CPU usage.
Type: boolean
queue.buffering.max.messagesP1 .. 10000000100000Maximum number of messages allowed on the producer queue.
Type: integer
queue.buffering.max.kbytesP1 .. 20971511048576Maximum total message size sum allowed on the producer queue. This property has higher priority than queue.buffering.max.messages.
Type: integer
queue.buffering.max.msP0 .. 9000000Delay in milliseconds to wait for messages in the producer queue to accumulate before constructing message batches (MessageSets) to transmit to brokers. A higher value allows larger and more effective (less overhead, improved compression) batches of messages to accumulate at the expense of increased message delivery latency.
Type: integer
linger.msPAlias for queue.buffering.max.ms
message.send.max.retriesP0 .. 100000002How many times to retry sending a failing MessageSet. Note: retrying may cause reordering.
Type: integer
retriesPAlias for message.send.max.retries
retry.backoff.msP1 .. 300000100The backoff time in milliseconds before retrying a protocol request.
Type: integer
queue.buffering.backpressure.thresholdP1 .. 10000001The threshold of outstanding not yet transmitted broker requests needed to backpressure the producer’s message accumulator. If the number of not yet transmitted requests equals or exceeds this number, produce request creation that would have otherwise been triggered (for example, in accordance with linger.ms) will be delayed. A lower number yields larger and more effective batches. A higher value can improve latency when using compression on slow machines.
Type: integer
compression.codecPnone, gzip, snappy, lz4nonecompression codec to use for compressing message sets. This is the default value for all topics, may be overriden by the topic configuration property compression.codec.
Type: enum value
compression.typePAlias for compression.codec
batch.num.messagesP1 .. 100000010000Maximum number of messages batched in one MessageSet. The total MessageSet size is also limited by message.max.bytes.
Type: integer
delivery.report.only.errorPtrue, falsefalseOnly provide delivery reports for failed messages.
Type: boolean
dr_cbPDelivery report callback (set with rd_kafka_conf_set_dr_cb())
Type: pointer
dr_msg_cbPDelivery report callback (set with rd_kafka_conf_set_dr_msg_cb())
Type: pointer

Topic configuration properties

PropertyC/PRangeDefaultDescription
request.required.acksP-1 .. 10001This field indicates how many acknowledgements the leader broker must receive from ISR brokers before responding to the request: 0=Broker does not send any response/ack to client, 1=Only the leader broker will need to ack the message, -1 or all=broker will block until message is committed by all in sync replicas (ISRs) or broker’s min.insync.replicas setting before sending response.
Type: integer
acksPAlias for request.required.acks
request.timeout.msP1 .. 9000005000The ack timeout of the producer request in milliseconds. This value is only enforced by the broker and relies on request.required.acks being != 0.
Type: integer
message.timeout.msP0 .. 900000300000Local message timeout. This value is only enforced locally and limits the time a produced message waits for successful delivery. A time of 0 is infinite. This is the maximum time librdkafka may use to deliver a message (including retries). Delivery error occurs when either the retry count or the message timeout are exceeded.
Type: integer
queuing.strategyPfifo, lifofifoProducer queuing strategy. FIFO preserves produce ordering, while LIFO prioritizes new messages. WARNING: lifo is experimental and subject to change or removal.
Type: enum value
produce.offset.reportPtrue, falsefalseReport offset of produced message back to application. The application must be use the dr_msg_cb to retrieve the offset from rd_kafka_message_t.offset.
Type: boolean
partitionerPconsistent_randomPartitioner: random - random distribution, consistent - CRC32 hash of key (Empty and NULL keys are mapped to single partition), consistent_random - CRC32 hash of key (Empty and NULL keys are randomly partitioned), murmur2 - Java Producer compatible Murmur2 hash of key (NULL keys are mapped to single partition), murmur2_random - Java Producer compatible Murmur2 hash of key (NULL keys are randomly partitioned. This is functionally equivalent to the default partitioner in the Java Producer.).
Type: string
partitioner_cbPCustom partitioner callback (set with rd_kafka_topic_conf_set_partitioner_cb())
Type: pointer
msg_order_cmpPMessage queue ordering comparator (set with rd_kafka_topic_conf_set_msg_order_cmp()). Also see queuing.strategy.
Type: pointer
opaque*Application opaque (set with rd_kafka_topic_conf_set_opaque())
Type: pointer
compression.codecPnone, gzip, snappy, lz4, inheritinheritCompression codec to use for compressing message sets. inherit = inherit global compression.codec configuration.
Type: enum value
compression.typePAlias for compression.codec
compression.levelP-1 .. 12-1Compression level parameter for algorithm selected by configuration property compression.codec. Higher values will result in better compression at the cost of more CPU usage. Usable range is algorithm-dependent: [0-9] for gzip; [0-12] for lz4; only 0 for snappy; -1 = codec-dependent default compression level.
Type: integer
auto.commit.enableCtrue, falsetrueIf true, periodically commit offset of the last message handed to the application. This committed offset will be used when the process restarts to pick up where it left off. If false, the application will have to call rd_kafka_offset_store() to store an offset (optional). NOTE: This property should only be used with the simple legacy consumer, when using the high-level KafkaConsumer the global enable.auto.commit property must be used instead. NOTE: There is currently no zookeeper integration, offsets will be written to broker or local file according to offset.store.method.
Type: boolean
enable.auto.commitCAlias for auto.commit.enable
auto.commit.interval.msC10 .. 8640000060000The frequency in milliseconds that the consumer offsets are committed (written) to offset storage. This setting is used by the low-level legacy consumer.
Type: integer
auto.offset.resetCsmallest, earliest, beginning, largest, latest, end, errorlargestAction to take when there is no initial offset in offset store or the desired offset is out of range: ‘smallest’,’earliest’ - automatically reset the offset to the smallest offset, ‘largest’,’latest’ - automatically reset the offset to the largest offset, ‘error’ - trigger an error which is retrieved by consuming messages and checking ‘message->err’.
Type: enum value
offset.store.pathC.Path to local file for storing offsets. If the path is a directory a filename will be automatically generated in that directory based on the topic and partition.
Type: string
offset.store.sync.interval.msC-1 .. 86400000-1fsync() interval for the offset file, in milliseconds. Use -1 to disable syncing, and 0 for immediate sync after each write.
Type: integer
offset.store.methodCfile, brokerbrokerOffset commit store method: ‘file’ - local file store (offset.store.path, et.al), ‘broker’ - broker commit store (requires “group.id” to be configured and Apache Kafka 0.8.2 or later on the broker.).
Type: enum value
consume.callback.max.messagesC0 .. 10000000Maximum number of messages to dispatch in one rd_kafka_consume_callback*() call (0 = unlimited)
Type: integer

C/P legend: C = Consumer, P = Producer, * = both

声明:本文内容由网友自发贡献,不代表【wpsshop博客】立场,版权归原作者所有,本站不承担相应法律责任。如您发现有侵权的内容,请联系我们。转载请注明出处:https://www.wpsshop.cn/w/正经夜光杯/article/detail/813225
推荐阅读
相关标签
  

闽ICP备14008679号