So this was a fun problem, ElasticSearch on AWS in it’s current form is completely public and not secured within my VPC. It appears that first you have to put in a deny statement to lock off access and then you add in the roles / users that have access. But once you do this you now have a box that can’t be accessed by CURL… very annoying. So now you need to write a python (or language of your choice) script to send a signed request. Here’s the code I’m working on:
#!/usr/local/bin/python import requests import json from requests_aws_sign import AWSV4Sign from boto3 import session from elasticsearch import Elasticsearch, RequestsHttpConnection import sys, getopt host = '' json_map = '' index = '' try: opts, args = getopt.getopt(sys.argv[1:],"h",["host=","json_map=","index="]) except getopt.GetoptError: print '--host <ES HOST> --index <index name> --json_map <ES Mapping>' sys.exit(2) for opt, arg in opts: if opt == '-h': print '--host <ES HOST> --index <index name> --json_map <ES Mapping>' sys.exit() elif opt == "--host": es_host = arg print es_host elif opt == "--json_map": json_map = arg print json_map elif opt == "--index": index_name = arg print index_name with open(json_map) as data_file: request_body = json.load(data_file) # Establish credentials session = session.Session() credentials = session.get_credentials() region = session.region_name or 'us-west-2' # Elasticsearch settings service = 'es' auth = AWSV4Sign(credentials, region, service) es_client = Elasticsearch(host=es_host, port=443, connection_class=RequestsHttpConnection, http_auth=auth, use_ssl=True, verify_ssl=True) # print es_client.info() try: res = es_client.indices.create(index=index_name,body=request_body) print(" response: '%s'" % (res)) except Exception as e: print("Not working...") print(e)
My developers can now commit their JSON to git to create their index mappings and it can be auto created by Jenkins.
=== UPDATE ===
Doesn’t really work, it appeared that I had a solution but finding that it’s 90% there.. Leaving this post here for now in hopes that it might be useful at some point.