Produce and consume Kafka messages over HTTPS with OVH Queue DBaaS
This tutorial assumes you already have an OVH account and have subscribed to the Queue DBaaS.
If not, have a look at our Getting Started guide that will also introduce you to Kafka
Why HTTP(S)?
Kafka protocol is built over TCP for multiple reasons, including performance and message ordering.
However we realized that for some use cases, Kafka clients usage can be quite complex since many parameters have to be taken into consideration:
- Number of partitions and partition leaders
- Offset commit
- Consumer groups
According to your programming language of choice, Kafka clients can be more or less efficient and easy to use.
Sometimes what you need is a simpler queue platform that you can use with any programming language. HTTP is the de-facto standard for that usage.
Queue DBaaS platform implements a simple HTTP layer that allows you to:
- Produce one or many messages to your topic with a POST request
- Consume one or many messages from your topic with a GET request
Moreover, it comes with HTTPS support, for free.
Usage
Pre-requisite: Know your region
According to how you created your Queue DBaaS app, you chose one of the available regions, which can be retrieved
on your Sunrise Manager. Here we are interested in the HTTPS URL.
Example:
https://kafka.p1.sbg.queue.ovh.net
Authentication
As with traditional Queue DBaaS usage with Kafka Clients, you will have to use:
- an SASL username (eg.
SASL_USERNAME=my-app.admin) - an SASL password (eg.
SASL_PASSWORD=aRT3u7R2TuRzhMmnLl) - a topic that your user is authorized to access (eg.
TOPIC=my-app.my-topic)
See our Getting Started
and SASL guides for more information.
User authentication is made with a standard HTTP basic auth in which you must provide your SASL username and password:
$SASL_USERNAME:$SASL_PASSWORD
POST messages
You can post one or multiple messages, using curl or HTTP clients in any language:
curl -u $SASL_USERNAME:$SASL_PASSWORD
-XPOST https://kafka.p1.sbg.queue.ovh.net/topic/$TOPIC
-d '[{"Value": "first message"},
{"Value":"second message"}]'
Please note in the example above that your message payload must be a JSON array of objects containing your message as a string in the “Value” field.
If this data model does not suit you, you can also post a message in raw format (accepts JSON, string, arrays, bytes…):
curl -u $SASL_USERNAME:$SASL_PASSWORD
-XPOST https://kafka.p1.sbg.queue.ovh.net/topic/$TOPIC?format=raw
-d '{"my": {"complex": ["json", "here"]}}'
GET messages
You can retrieve one or multiple messages, using curl or HTTP clients in any language:
curl -u $SASL_USERNAME:$SASL_PASSWORD
https://kafka.p1.sbg.queue.ovh.net/topic/$TOPIC[?limit=2]
The optional limit parameter allows you to specify how many messages you want to retrieve.
Offset management
Offsets are managed automatically at each GET request. It means that each message will be read only once.
We plan to support manual offset and consumer group specification later on.
Message type
As seen in the examples above, messages are sent through a JSON object and must be strings.
This does not prevent you from using other data types, by encoding your binary data as base64 for example.
Performance concerns
Despite the ease-of-use, using HTTPS instead of TCP implies a performance hit.
To put it simply: if you need a very high read/write throughput, use a normal TCP-based Kafka Client.
As an example, here are numbers on producing and consuming “hello world” messages:
- via a TCP Kafka Client: 200k+ messages per second
- via the HTTPS endpoint: 5-10k messages per second
Note that the number of messages you send or receive with a single HTTP request has an impact on performance: don’t hesitate to batch messages together.
Contact and feedbacks
Please feel free to contact us for any question, feedback, or improvement that may come to your mind!
