AWS SigV4 support for OpenSearch clients
OpenSearch clients now support the ability to sign requests using AWS Signature V4. This has been a community request for a while, and we’re happy to announce that we have completed work across all clients, in collaboration with external contributors. Signing requests using native clients has been an essential requirement for accessing the Amazon OpenSearch Service on AWS using fine grained access controls. Having native SigV4 support in clients avoids the need to use cURL requests and other workarounds.
Setting up the managed service to use fine-grained access control
Be sure to update your access control type to an AWS Identity and Access Management (IAM) role/user. Do not use the master user role. In the following image, the IAM role allows access to the specific OpenSearch domain that is selected:
Alternatively, you can set a domain-level access policy without using fine-grained access. Ensure that the IAM role you use has read/write access to the domain.
Creating a client connection using SigV4 signing
Before you begin, ensure that you have AWS credentials set up on your machine. AWS credentials can be stored in ~/.aws/credentials
or set as AWS_
environment variables, and contain an access key, a secret key and an optional session token, that allow you to authenticate with AWS resources using IAM.
Creating a client connection in Java
import java.io.IOException;
import org.opensearch.client.opensearch.OpenSearchClient;
import org.opensearch.client.opensearch.core.InfoResponse;
import org.opensearch.client.transport.aws.AwsSdk2Transport;
import org.opensearch.client.transport.aws.AwsSdk2TransportOptions;
import software.amazon.awssdk.http.SdkHttpClient;
import software.amazon.awssdk.http.apache.ApacheHttpClient;
import software.amazon.awssdk.regions.Region;
public static void main(final String[] args) throws IOException {
SdkHttpClient httpClient = ApacheHttpClient.builder().build();
try {
OpenSearchClient client = new OpenSearchClient(
new AwsSdk2Transport(
httpClient,
"search-xxx.region.es.amazonaws.com",
Region.US_WEST_2,
AwsSdk2TransportOptions.builder().build()
)
);
InfoResponse info = client.info();
System.out.println(info.version().distribution() + ": " + info.version().number());
} finally {
httpClient.close();
}
}
Creating a client connection in Python
The Python client requires you to have boto3
installed. Make sure to update the connection_class
to use RequestsHttpConnection
.
from urllib.parse import urlparse
from boto3 import Session
from opensearchpy import AWSV4SignerAuth, OpenSearch, RequestsHttpConnection
url = urlparse("https://search-xxx.region.es.amazonaws.com")
region = 'us-east-1'
credentials = Session().get_credentials()
auth = AWSV4SignerAuth(credentials, region)
client = OpenSearch(
hosts=[{
'host': url.netloc,
'port': url.port or 443
}],
http_auth=auth,
use_ssl=True,
verify_certs=True,
connection_class=RequestsHttpConnection
)
info = client.info()
print(f"{info['version']['distribution']}: {info['version']['number']}")
Creating a client connection in JavaScript
The JavaScript client requires you to have aws-sdk
installed. Depending on which version of the SDK you are using, initialize the client appropriately as shown in the following code segments.
Using AWS V2 SDK
const AWS = require('aws-sdk');
const { Client } = require('@opensearch-project/opensearch');
const { AwsSigv4Signer } = require('@opensearch-project/opensearch/aws');
const client = new Client({
...AwsSigv4Signer({
region: 'us-east-1',
getCredentials: () =>
new Promise((resolve, reject) => {
AWS.config.getCredentials((err, credentials) => {
if (err) {
reject(err);
} else {
resolve(credentials);
}
});
}),
}),
node: "https://search-xxx.region.es.amazonaws.com"
});
Using AWS V3 SDK
const { defaultProvider } = require("@aws-sdk/credential-provider-node");
const { Client } = require('@opensearch-project/opensearch');
const { AwsSigv4Signer } = require('@opensearch-project/opensearch/aws');
async function main() {
const client = new Client({
...AwsSigv4Signer({
region: "us-east-1",
getCredentials: () => {
const credentialsProvider = defaultProvider();
return credentialsProvider();
},
}),
node: "https://search-xxx.region.es.amazonaws.com"
});
var info = await client.info();
var version = info.body.version
console.log(version.distribution + ": " + version.number);
}
main();
Creating a client connection in Ruby
The opensearch-aws-sigv4 gem provides the OpenSearch::Aws::Sigv4Client
class, which has all the features of OpenSearch::Client
. The only difference between these two clients is that OpenSearch::Aws::Sigv4Client
requires an instance of Aws::Sigv4::Signer
during instantiation to authenticate with AWS.
require 'opensearch-aws-sigv4'
require 'aws-sigv4'
signer = Aws::Sigv4::Signer.new(
service: 'es',
region: 'us-east-1',
access_key_id: '...',
secret_access_key: '...',
session_token: '...'
)
client = OpenSearch::Aws::Sigv4Client.new({
host: "https://search-xxx.region.es.amazonaws.com",
log: false
}, signer)
info = client.info
puts info['version']['distribution'] + ': ' + info['version']['number']
Creating a client connection in .NET
All required request signing is handled by the AwsSigV4HttpConnection
implementation. By default, AwsSigV4HttpConnection
uses the .NET AWS SDK’s default credentials provider to acquire credentials from the environment. However, you may opt to pass in your own credentials provider, for example, to assume a role. Refer to the OpenSearch.Net User Guide for complete instructions.
using OpenSearch.Client;
using OpenSearch.Net.Auth.AwsSigV4;
namespace Application
{
class Program
{
static void Main(string[] args)
{
var endpoint = new Uri("https://search-xxx.region.es.amazonaws.com");
var connection = new AwsSigV4HttpConnection();
var config = new ConnectionSettings(endpoint, connection);
var client = new OpenSearchClient(config);
Console.WriteLine($"{client.RootNodeInfo().Version.Distribution}: {client.RootNodeInfo().Version.Number}");
}
}
}
Creating a client connection in Rust
Request signing is configured using the Credentials::AwsSigV4
enum variant or its helper conversion from an AWS SDK configuration. See aws-config for other AWS credentials provider implementations, for example, to assume a role.
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
use std::{convert::TryInto, env, thread, time};
use serde_json::Value;
use opensearch::{
http::transport::{SingleNodeConnectionPool, TransportBuilder},
OpenSearch,
};
use url::Url;
let url = Url::parse("https://search-xxx.region.es.amazonaws.com");
let conn_pool = SingleNodeConnectionPool::new(url?);
let aws_config = aws_config::load_from_env().await.clone();
let transport = TransportBuilder::new(conn_pool)
.auth(aws_config.clone().try_into()?)
.build()?;
let client = OpenSearch::new(transport);
let info: Value = client.info().send().await?.json().await?;
println!(
"{}: {}",
info["version"]["distribution"].as_str().unwrap(),
info["version"]["number"].as_str().unwrap()
);
Ok(())
}
Creating a client connection in PHP
The PHP client uses the setSigV4CredentialProvider
attribute to assume credentials from the the local credential store. Use the setSigV4Region
attribute to set the AWS Region.
<?php
require_once __DIR__ . '/vendor/autoload.php';
$client = (new \OpenSearch\ClientBuilder())
->setHosts(["https://search-xxx.region.es.amazonaws.com"])
->setSigV4Region("us-east-1")
->setSigV4CredentialProvider(true)
->build();
$info = $client->info();
echo "{$info['version']['distribution']}: {$info['version']['number']}\n";
Creating a client connection in Go
package main
import (
"context"
"encoding/json"
"fmt"
"log"
"strings"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/opensearch-project/opensearch-go/v2"
requestsigner "github.com/opensearch-project/opensearch-go/v2/signer/awsv2"
)
func main() {
ctx := context.Background()
cfg, _ := config.LoadDefaultConfig(ctx)
signer, _ := requestsigner.NewSigner(cfg)
endpoint := "https://search-xxx.region.es.amazonaws.com"
client, _ := opensearch.NewClient(opensearch.Config{
Addresses: []string{endpoint},
Signer: signer,
})
if info, err := client.Info(); err != nil {
log.Fatal("info", err)
} else {
var r map[string]interface{}
json.NewDecoder(info.Body).Decode(&r)
version := r["version"].(map[string]interface{})
fmt.Printf("%s: %s\n", version["distribution"], version["number"])
}
}
Use with Amazon OpenSearch Serverless (preview)
Refer to this article for information about how to use clients with Amazon OpenSearch Serverless.
Summary
You can now sign your requests natively using the client APIs instead of workarounds. We’re continuing to work on improving the capabilities of SigV4 in clients with scenarios like asynchronous connections, compressed requests, and connection pooling support, and we welcome your pull requests and feedback in the form of issues on GitHub.