Manage OpenSearch® log integration

Aiven provides a service integration that allows you to send your logs from several services, such as Aiven for Apache Kafka®, PostgreSQL®, Apache Cassandra®, OpenSearch®, Caching, and Grafana®, to Aiven for OpenSearch®, allowing you to use OpenSearch to gain more insight and control over your logs.

tip

See this video tutorial for an end-to-end example of how to enable your Aiven for OpenSearch® log integration.

Prerequisites

Console

• User profile in the Aiven Console
• Source service: Aiven-managed service producing logs to be sent to Aiven for OpenSearch
• Destination service: Aiven for OpenSearch service that receives the logs

API

• Personal token for use with the Aiven API, CLI, Terraform, or other applications
• Aiven API
• Source service: Aiven-managed service producing logs to be sent to Aiven for OpenSearch
• Destination service: Aiven for OpenSearch service that receives the logs

CLI

• Personal token for use with the Aiven API, CLI, Terraform, or other applications
• Aiven CLI
• Source service: Aiven-managed service producing logs to be sent to Aiven for OpenSearch
• Destination service: Aiven for OpenSearch service that receives the logs

Terraform

• Personal token for use with the Aiven API, CLI, Terraform, or other applications
• Aiven Provider for Terraform
• Source service: Aiven-managed service producing logs to be sent to Aiven for OpenSearch
• Destination service: Aiven for OpenSearch service that receives the logs

Kubernetes

• Personal token for use with the Aiven API, CLI, Terraform, or other applications
• Aiven Operator for Kubernetes®
• Source service: Aiven-managed service producing logs to be sent to Aiven for OpenSearch
• Destination service: Aiven for OpenSearch service that receives the logs

Enable log integration

Enable logs integration to send logs from your service to Aiven for OpenSearch®:

Console

1. Log in to the Aiven Console and go to the service that produces the logs to be sent to Aiven for OpenSearch.

2. Click Logs in the sidebar.

3. On the Logs page, click Enable logs integration.

4. In the Logs integration window, select an existing Aiven for OpenSearch service or create one, and click Continue.

   note

   If you choose to select an existing service and you are a member of more than one Aiven project with operator or admin access rights, select a project before selecting an Aiven for OpenSearch service.

5. In the Configure logs integration window, set up the index prefix and index retention limit parameters, and click Enable.

   note

   To effectively disable the index retention limit, set it to its maximum value of 10000 days.

API

Call the ServiceIntegrationCreate endpoint to enable log integration:

curl --request POST \
  --url https://api.aiven.io/v1/project/PROJECT_NAME/integration \
  --header "Authorization: Bearer API_TOKEN" \
  --header "Content-Type: application/json" \
  --data-raw '{
  "integration_type": "logs",
  "source_service": "SOURCE_SERVICE_NAME",
  "dest_service": "OPENSEARCH_SERVICE_NAME",
  "user_config": {
    "elasticsearch_index_prefix": "INDEX_PREFIX",
    "elasticsearch_index_days_max": INDEX_RETENTION_DAYS
  }
}'

Parameters:

• PROJECT_NAME: Your Aiven project name
• API_TOKEN: Your personal token
• SOURCE_SERVICE_NAME: The service producing logs
• OPENSEARCH_SERVICE_NAME: Your Aiven for OpenSearch service name
• INDEX_PREFIX: Prefix for the index name
• INDEX_RETENTION_DAYS: Number of days to keep logs

Example:

curl --request POST \
  --url https://api.aiven.io/v1/project/dev-sandbox/integration \
  --header "Authorization: Bearer 123abc456def789ghi" \
  --header "Content-Type: application/json" \
  --data-raw '{
  "integration_type": "logs",
  "source_service": "my-postgresql",
  "dest_service": "my-opensearch",
  "user_config": {
    "elasticsearch_index_prefix": "logs",
    "elasticsearch_index_days_max": 3
  }
}'

CLI

Use the avn service integration-create command to enable log integration:

avn service integration-create \
  --project PROJECT_NAME \
  --source-service SOURCE_SERVICE_NAME \
  --dest-service OPENSEARCH_SERVICE_NAME \
  --integration-type logs \
  -c elasticsearch_index_prefix=INDEX_PREFIX \
  -c elasticsearch_index_days_max=INDEX_RETENTION_DAYS

Parameters:

• PROJECT_NAME: Your Aiven project name
• SOURCE_SERVICE_NAME: The service producing logs
• OPENSEARCH_SERVICE_NAME: Your Aiven for OpenSearch service name
• INDEX_PREFIX: Prefix for the index name
• INDEX_RETENTION_DAYS: Number of days to keep logs

Example:

avn service integration-create \
  --project my-project \
  --source-service my-kafka-service \
  --dest-service my-opensearch \
  --integration-type logs \
  -c elasticsearch_index_prefix=logs \
  -c elasticsearch_index_days_max=7

Terraform

Add a aiven_service_integration resource to enable log integration:

resource "aiven_service_integration" "logs_integration" {
  project                  = "PROJECT_NAME"
  integration_type         = "logs"
  source_service_name      = "SOURCE_SERVICE_NAME"
  destination_service_name = "OPENSEARCH_SERVICE_NAME"

  logs_user_config {
    elasticsearch_index_prefix   = "INDEX_PREFIX"
    elasticsearch_index_days_max = INDEX_RETENTION_DAYS
  }
}

Parameters:

• PROJECT_NAME: Your Aiven project name
• SOURCE_SERVICE_NAME: The service producing logs
• OPENSEARCH_SERVICE_NAME: Your Aiven for OpenSearch service name
• INDEX_PREFIX: Prefix for the index name, for example, logs
• INDEX_RETENTION_DAYS: Number of days to keep logs, for example, 3

Run terraform init, terraform plan, and terraform apply.

For more information, see the aiven-service-integration resource documentation.

Kubernetes

Add a ServiceIntegration resource to enable log integration:

apiVersion: aiven.io/v1alpha1
kind: ServiceIntegration
metadata:
  name: INTEGRATION_RESOURCE_NAME
spec:
  authSecretRef:
    name: aiven-token
    key: TOKEN_NAME
  project: PROJECT_NAME
  integrationType: logs
  sourceServiceName: SOURCE_SERVICE_NAME
  destinationServiceName: OPENSEARCH_SERVICE_NAME
  logs:
    elasticsearch_index_prefix: INDEX_PREFIX
    elasticsearch_index_days_max: INDEX_RETENTION_DAYS

Apply the resource with kubectl apply -f FILE_NAME.yaml.

For more information, see the ServiceIntegration resource documentation.

Configure log integration

There are two parameters that you can adjust when integrating logs to your OpenSearch service:

• index prefix, specifies the prefix part of the index name
• index retention limit, number of days to preserve the daily indexes

warning

The service's logs are sent from the selected service to your OpenSearch cluster. When the index retention limit is reached, those indexes are deleted from the OpenSearch cluster.

You can change the configuration of the index prefix and index retention limit after the integration is enabled.

Console

1. Log in to the Aiven Console, and go to your Aiven for OpenSearch service.
2. Click Integrations in the sidebar.
3. On the Integrations page, find the integrated service to configure.
4. Click Actions > Edit.
5. After updating index prefix or index retention limit, click Edit.

API

1. Get INTEGRATION_ID by calling the ServiceGet endpoint:

   curl --request GET \
     --url https://api.aiven.io/v1/project/PROJECT_NAME/service/SERVICE_NAME \
     --header "Authorization: Bearer API_TOKEN"

   In the output, under service_integrations, find your integration and its service_integration_id.

2. Call the ServiceIntegrationUpdate endpoint to update the log integration configuration:

   curl --request PUT \
     --url https://api.aiven.io/v1/project/PROJECT_NAME/integration/INTEGRATION_ID \
     --header "Authorization: Bearer API_TOKEN" \
     --header "Content-Type: application/json" \
     --data-raw '{
     "user_config": {
       "elasticsearch_index_prefix": "UPDATED_INDEX_PREFIX",
       "elasticsearch_index_days_max": UPDATED_INDEX_RETENTION_DAYS
     }
   }'

   Parameters:

   • PROJECT_NAME: Your Aiven project name
   • INTEGRATION_ID: The integration ID
   • API_TOKEN: Your personal token
   • UPDATED_INDEX_PREFIX: Updated prefix for the index name
   • UPDATED_INDEX_RETENTION_DAYS: Updated number of days to keep logs

   Example:

   curl --request PUT \
     --url https://api.aiven.io/v1/project/my-project/integration/123abc456def789ghi-123abc456def789ghi \
     --header "Authorization: Bearer 123abc456def789ghi" \
     --header "Content-Type: application/json" \
     --data-raw '{
     "user_config": {
       "elasticsearch_index_prefix": "new",
       "elasticsearch_index_days_max": 100
     }
   }'

CLI

1. Get INTEGRATION_ID by running the avn service integration-list command:

   avn service integration-list SOURCE_SERVICE_NAME --project PROJECT_NAME

   Parameters:

   • PROJECT_NAME: Your Aiven project name
   • SOURCE_SERVICE_NAME: The service producing logs

   In the output table, find SERVICE_INTEGRATION_ID for the integrated services listed in columns SOURCE and DEST.

2. Use the avn service integration-update command to update the log integration configuration:

   avn service integration-update INTEGRATION_ID \
     --project PROJECT_NAME \
     -c elasticsearch_index_prefix=UPDATED_INDEX_PREFIX \
     -c elasticsearch_index_days_max=UPDATED_INDEX_RETENTION_DAYS

   Parameters:

   • INTEGRATION_ID: The integration ID
   • PROJECT_NAME: Your Aiven project name
   • UPDATED_INDEX_PREFIX: Updated prefix for the index name
   • UPDATED_INDEX_RETENTION_DAYS: Updated number of days to keep logs

   Example:

   avn service integration-update 123abc456def789ghi-123abc456def789ghi \
     --project my-project \
     -c elasticsearch_index_prefix=new \
     -c elasticsearch_index_days_max=100

Terraform

Update the logs_user_config block in your aiven_service_integration resource:

resource "aiven_service_integration" "logs_integration" {
  project                  = "PROJECT_NAME"
  integration_type         = "logs"
  source_service_name      = "SOURCE_SERVICE_NAME"
  destination_service_name = "OPENSEARCH_SERVICE_NAME"

  logs_user_config {
    elasticsearch_index_prefix   = "UPDATED_INDEX_PREFIX"
    elasticsearch_index_days_max = UPDATED_INDEX_RETENTION_DAYS
  }
}

Parameters:

• PROJECT_NAME: Your Aiven project name
• SOURCE_SERVICE_NAME: The service producing logs
• OPENSEARCH_SERVICE_NAME: Your Aiven for OpenSearch service name
• UPDATED_INDEX_PREFIX: Prefix for the index name, for example, logs
• UPDATED_INDEX_RETENTION_DAYS: Number of days to keep logs, for example, 3

Run terraform apply to apply the changes.

Kubernetes

Update the logs section in your ServiceIntegration resource:

apiVersion: aiven.io/v1alpha1
kind: ServiceIntegration
metadata:
  name: INTEGRATION_RESOURCE_NAME
spec:
  authSecretRef:
    name: aiven-token
    key: TOKEN_NAME
  project: PROJECT_NAME
  integrationType: logs
  sourceServiceName: SOURCE_SERVICE_NAME
  destinationServiceName: OPENSEARCH_SERVICE_NAME
  logs:
    elasticsearch_index_prefix: UPDATED_INDEX_PREFIX
    elasticsearch_index_days_max: UPDATED_INDEX_RETENTION_DAYS

Apply the resource with kubectl apply -f FILE_NAME.yaml.

Disable logs integration

To stop sending logs from your service to Aiven for OpenSearch, disable the integration:

Console

1. Log in to the Aiven Console, and go to your integrated Aiven for OpenSearch service.
2. Click Integrations in the sidebar.
3. On the Integrations page, find the service sending its logs to your Aiven for OpenSearch service.
4. Click Actions > Disconnect.

API

1. Get INTEGRATION_ID by calling the ServiceGet endpoint:

   curl --request GET \
     --url https://api.aiven.io/v1/project/PROJECT_NAME/service/SERVICE_NAME \
     --header "Authorization: Bearer API_TOKEN"

   In the output, under service_integrations, find your integration and its service_integration_id.

2. Call the ServiceIntegrationDelete endpoint to delete the log integration:

   curl --request DELETE \
     --url https://api.aiven.io/v1/project/PROJECT_NAME/integration/INTEGRATION_ID \
     --header "Authorization: Bearer API_TOKEN"

   Parameters:

   • PROJECT_NAME: Your Aiven project name
   • INTEGRATION_ID: The integration ID
   • API_TOKEN: Your personal token

   Example:

   curl --request DELETE \
     --url https://api.aiven.io/v1/project/dev-sandbox/integration/123abc456def789ghi-123abc456def789ghi \
     --header "Authorization: Bearer 123abc456def789ghi"

CLI

1. Get INTEGRATION_ID by running the avn service integration-list command:

   avn service integration-list SOURCE_SERVICE_NAME --project PROJECT_NAME

   Parameters:

   • PROJECT_NAME: Your Aiven project name
   • SOURCE_SERVICE_NAME: The service producing logs

   In the output table, find SERVICE_INTEGRATION_ID for the integrated services listed in columns SOURCE and DEST.

2. Use the avn service integration-delete command to delete the log integration:

   avn service integration-delete INTEGRATION_ID \
     --project PROJECT_NAME

   Parameters:

   • INTEGRATION_ID: The integration ID
   • PROJECT_NAME: Your Aiven project name

   Example:

   avn service integration-delete 123abc456def789ghi-123abc456def789ghi \
     --project my-project

Terraform

Remove the aiven_service_integration resource from your Terraform configuration:

# Remove or comment out this resource
# resource "aiven_service_integration" "logs_integration" {
#   project                  = "PROJECT_NAME"
#   integration_type         = "logs"
#   source_service_name      = "SOURCE_SERVICE_NAME"
#   destination_service_name = "OPENSEARCH_SERVICE_NAME"
#
#   logs_user_config {
#     elasticsearch_index_prefix   = "INDEX_PREFIX"
#     elasticsearch_index_days_max = INDEX_RETENTION_DAYS
#   }
# }

Run terraform apply to delete the integration.

Kubernetes

Delete the ServiceIntegration resource:

kubectl delete serviceintegration INTEGRATION_RESOURCE_NAME

or

kubectl delete -f FILE_NAME.yaml