Getting Started with Amazon S3 Vectors
Build a semantic search system using AWS's new serverless vector storage. Store millions of embeddings in S3 with sub-second query times and serverless pricing.
On this page (13 sections)
- When to Use S3 Vectors
- What We’re Building
- Step 1: Set Up the S3 Vectors Client
- Step 2: Create a Vector Bucket
- Step 3: Create a Vector Index
- Step 4: Generate Embeddings with Bedrock
- Step 5: Store Vectors with Metadata
- Step 6: Query for Similar Vectors
- Step 7: Filter by Metadata
- Pricing Considerations
- Cleanup
- Full Code
- What’s Next
AWS launched S3 Vectors, native vector storage built directly into S3. No servers to provision, no clusters to manage, no indexes to tune. You store vectors like objects and query them with sub-second latency.
The pitch is compelling: serverless pricing with no cluster management, support for 2 billion vectors per index, and native Bedrock integration. But when should you actually use it?
Before Setting Sail
- AWS account with Bedrock access
- Python 3.9+
- boto3 1.42+ (pip install boto3 --upgrade)
- IAM permissions for S3 Vectors and Bedrock
When to Use S3 Vectors
S3 Vectors isn’t trying to replace your existing vector database. It’s optimized for a specific pattern:
| Use Case | Best Choice |
|---|---|
| Real-time search (<50ms) | OpenSearch, Pinecone |
| High query volume (thousands/min) | OpenSearch, Pinecone |
| Large archive, occasional queries | S3 Vectors |
| RAG with Bedrock Knowledge Bases | S3 Vectors |
| Cost-sensitive workloads | S3 Vectors |
The sweet spot is large datasets with infrequent queries, such as document archives, historical embeddings, or RAG knowledge bases where you aren’t hammering the index constantly.
Captain's Note
S3 Vectors delivers ~100ms latency for warm queries and sub-second for cold queries. If you need single-digit milliseconds, stick with an in-memory vector database.
What We’re Building
We’ll build a movie recommendation system that:
- Creates a vector bucket and index in S3
- Generates embeddings with Amazon Titan
- Stores movie descriptions with metadata
- Queries for similar movies with metadata filtering
By the end, you’ll have a working semantic search system running entirely on serverless AWS infrastructure.
Step 1: Set Up the S3 Vectors Client
S3 Vectors uses a separate boto3 client from regular S3:
import boto3
import json
# S3 Vectors has its own client (not the regular s3 client)
s3vectors = boto3.client("s3vectors", region_name="us-east-1")
bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")
Rough Waters Ahead
S3 Vectors is available in us-east-1, us-west-2, and eu-west-1 at launch. Check the AWS documentation for current region availability.
Step 2: Create a Vector Bucket
Vector buckets are a new S3 bucket type specifically designed for vector storage:
# Create a vector bucket
bucket_name = "movie-embeddings-demo"
response = s3vectors.create_vector_bucket(
vectorBucketName=bucket_name
)
print(f"Created bucket: {response['vectorBucketArn']}")
Created bucket: arn:aws:s3vectors:us-east-1:123456789012:vector-bucket/movie-embeddings-demo
Captain's Note
Vector buckets automatically have Block Public Access enabled, preventing accidental exposure of your embeddings to the internet.
Step 3: Create a Vector Index
Within a bucket, you create indexes to organize your vectors. Each index specifies the embedding dimension and distance metric:
# Create an index for movie embeddings
# Titan Text Embeddings V2 produces 1024-dimensional vectors
index_name = "movies"
response = s3vectors.create_index(
vectorBucketName=bucket_name,
indexName=index_name,
dimension=1024,
distanceMetric="cosine"
)
print(f"Created index: {response['indexArn']}")
Created index: arn:aws:s3vectors:us-east-1:123456789012:vector-bucket/movie-embeddings-demo/index/movies
Key parameters:
| Parameter | Options | Notes |
|---|---|---|
dimension | 1-16000 | Must match your embedding model |
distanceMetric | cosine, euclidean, dotProduct | Cosine is most common for text |
Step 4: Generate Embeddings with Bedrock
Now let’s create embeddings for some movie descriptions using Amazon Titan:
def generate_embedding(text: str) -> list[float]:
"""Generate an embedding using Amazon Titan Text Embeddings V2."""
response = bedrock.invoke_model(
modelId="amazon.titan-embed-text-v2:0",
body=json.dumps({"inputText": text})
)
response_body = json.loads(response["body"].read())
return response_body["embedding"]
# Sample movie data
movies = [
{
"key": "star-wars",
"title": "Star Wars: A New Hope",
"description": "A farm boy joins rebels to fight an evil empire in space.",
"genre": "scifi",
"year": 1977
},
{
"key": "jurassic-park",
"title": "Jurassic Park",
"description": "Scientists create dinosaurs in a theme park that goes terribly wrong.",
"genre": "scifi",
"year": 1993
},
{
"key": "finding-nemo",
"title": "Finding Nemo",
"description": "A clownfish father searches the ocean to rescue his lost son.",
"genre": "family",
"year": 2003
},
{
"key": "the-matrix",
"title": "The Matrix",
"description": "A hacker discovers reality is a simulation controlled by machines.",
"genre": "scifi",
"year": 1999
},
{
"key": "inception",
"title": "Inception",
"description": "A thief enters dreams to plant ideas in people's minds.",
"genre": "scifi",
"year": 2010
}
]
# Generate embeddings for each movie
for movie in movies:
movie["embedding"] = generate_embedding(movie["description"])
print(f"Generated embedding for: {movie['title']}")
Generated embedding for: Star Wars: A New Hope Generated embedding for: Jurassic Park Generated embedding for: Finding Nemo Generated embedding for: The Matrix Generated embedding for: Inception
Step 5: Store Vectors with Metadata
Upload the vectors to S3 with their metadata. Metadata enables filtering during queries:
# Prepare vectors for upload
vectors = [
{
"key": movie["key"],
"data": {"float32": movie["embedding"]},
"metadata": {
"title": movie["title"],
"description": movie["description"],
"genre": movie["genre"],
"year": movie["year"]
}
}
for movie in movies
]
# Upload to S3 Vectors
response = s3vectors.put_vectors(
vectorBucketName=bucket_name,
indexName=index_name,
vectors=vectors
)
print(f"Uploaded {len(vectors)} vectors")
Uploaded 5 vectors
Metadata Types
S3 Vectors supports string, number, boolean, and list metadata types. All metadata is filterable by default; you can mark fields as non-filterable during index creation to exclude them from filter queries.
Step 6: Query for Similar Vectors
Now for semantic search: let’s find movies similar to a query:
def search_movies(query: str, top_k: int = 3) -> list[dict]:
"""Search for movies similar to the query."""
# Generate query embedding
query_embedding = generate_embedding(query)
# Query the index
response = s3vectors.query_vectors(
vectorBucketName=bucket_name,
indexName=index_name,
queryVector={"float32": query_embedding},
topK=top_k,
returnDistance=True,
returnMetadata=True
)
return response["vectors"]
# Search for space adventure movies
results = search_movies("epic space adventure with heroes fighting villains")
print("Search: 'epic space adventure with heroes fighting villains'\n")
for result in results:
meta = result["metadata"]
print(f" {meta['title']} ({meta['year']})")
print(f" Distance: {result['distance']:.4f}")
print(f" Genre: {meta['genre']}\n")
Search: ‘epic space adventure with heroes fighting villains’
Star Wars: A New Hope (1977) Distance: 0.3142 Genre: scifi
The Matrix (1999) Distance: 0.4521 Genre: scifi
Inception (2010) Distance: 0.5234 Genre: scifi
Star Wars is the top match for a space adventure query, right where we’d expect it.
Step 7: Filter by Metadata
Combine semantic search with metadata filters for more precise results:
# Search for scifi movies only
results = s3vectors.query_vectors(
vectorBucketName=bucket_name,
indexName=index_name,
queryVector={"float32": generate_embedding("family friendly adventure")},
topK=3,
filter={"genre": "scifi"}, # Only return scifi movies
returnDistance=True,
returnMetadata=True
)
print("Search: 'family friendly adventure' (filtered to scifi)\n")
for result in results["vectors"]:
meta = result["metadata"]
print(f" {meta['title']} - Distance: {result['distance']:.4f}")
Search: ‘family friendly adventure’ (filtered to scifi)
Jurassic Park - Distance: 0.4892 Star Wars: A New Hope - Distance: 0.5124 The Matrix - Distance: 0.6341
Notice that Finding Nemo (the actual best match for “family friendly”) is excluded because we filtered to genre: "scifi".
Rough Waters Ahead
Filtering on non-filterable metadata fields returns a 400 Bad Request error. Make sure your filter fields weren’t marked as non-filterable during index creation.
Pricing Considerations
S3 Vectors pricing has three components:
| Component | Cost | Notes |
|---|---|---|
| Storage | $0.06/GB/month | 1024-dim vector = 4KB |
| PUT operations | $0.20/GB | One-time ingestion cost |
| Queries | $2.50/million + data processing | Processing scales with index size |
Example costs for 1 million 1024-dimensional vectors:
- Storage: ~4GB × $0.06 = $0.24/month
- Ingestion: 4GB × $0.20 = $0.80 one-time
- 10K queries/month: ~$0.03/month
Compare this to a dedicated vector database instance running 24/7, and the savings are significant for low-query workloads.
Cost Optimization
S3 Vectors shines when you have millions of vectors but only query occasionally. For high-throughput workloads (thousands of queries per minute), the per-query costs can exceed a dedicated vector database.
Cleanup
Don’t forget to clean up resources when you’re done experimenting:
# Delete the index first
s3vectors.delete_index(
vectorBucketName=bucket_name,
indexName=index_name
)
print(f"Deleted index: {index_name}")
# Then delete the bucket
s3vectors.delete_vector_bucket(
vectorBucketName=bucket_name
)
print(f"Deleted bucket: {bucket_name}")
Full Code
Here’s the complete implementation in one file:
"""
S3 Vectors Movie Recommendation Demo
Requires: boto3 >= 1.42, AWS credentials with S3 Vectors and Bedrock access
"""
import boto3
import json
# Initialize clients
s3vectors = boto3.client("s3vectors", region_name="us-east-1")
bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")
BUCKET_NAME = "movie-embeddings-demo"
INDEX_NAME = "movies"
def generate_embedding(text: str) -> list[float]:
"""Generate embedding using Amazon Titan Text Embeddings V2."""
response = bedrock.invoke_model(
modelId="amazon.titan-embed-text-v2:0",
body=json.dumps({"inputText": text})
)
return json.loads(response["body"].read())["embedding"]
def setup_index():
"""Create vector bucket and index."""
# Create bucket
s3vectors.create_vector_bucket(vectorBucketName=BUCKET_NAME)
print(f"Created bucket: {BUCKET_NAME}")
# Create index (1024 dimensions for Titan V2)
s3vectors.create_index(
vectorBucketName=BUCKET_NAME,
indexName=INDEX_NAME,
dimension=1024,
distanceMetric="cosine"
)
print(f"Created index: {INDEX_NAME}")
def load_movies():
"""Load sample movie data with embeddings."""
movies = [
{"key": "star-wars", "title": "Star Wars: A New Hope",
"description": "A farm boy joins rebels to fight an evil empire in space.",
"genre": "scifi", "year": 1977},
{"key": "jurassic-park", "title": "Jurassic Park",
"description": "Scientists create dinosaurs in a theme park that goes terribly wrong.",
"genre": "scifi", "year": 1993},
{"key": "finding-nemo", "title": "Finding Nemo",
"description": "A clownfish father searches the ocean to rescue his lost son.",
"genre": "family", "year": 2003},
{"key": "the-matrix", "title": "The Matrix",
"description": "A hacker discovers reality is a simulation controlled by machines.",
"genre": "scifi", "year": 1999},
{"key": "inception", "title": "Inception",
"description": "A thief enters dreams to plant ideas in people's minds.",
"genre": "scifi", "year": 2010},
]
vectors = []
for movie in movies:
embedding = generate_embedding(movie["description"])
vectors.append({
"key": movie["key"],
"data": {"float32": embedding},
"metadata": {
"title": movie["title"],
"description": movie["description"],
"genre": movie["genre"],
"year": movie["year"]
}
})
print(f"Generated embedding: {movie['title']}")
s3vectors.put_vectors(
vectorBucketName=BUCKET_NAME,
indexName=INDEX_NAME,
vectors=vectors
)
print(f"Uploaded {len(vectors)} vectors")
def search(query: str, top_k: int = 3, genre_filter: str = None):
"""Semantic search with optional genre filter."""
query_embedding = generate_embedding(query)
params = {
"vectorBucketName": BUCKET_NAME,
"indexName": INDEX_NAME,
"queryVector": {"float32": query_embedding},
"topK": top_k,
"returnDistance": True,
"returnMetadata": True
}
if genre_filter:
params["filter"] = {"genre": genre_filter}
response = s3vectors.query_vectors(**params)
print(f"\nSearch: '{query}'" + (f" (genre: {genre_filter})" if genre_filter else ""))
for result in response["vectors"]:
meta = result["metadata"]
print(f" {meta['title']} ({meta['year']}) - Distance: {result['distance']:.4f}")
def cleanup():
"""Delete index and bucket."""
s3vectors.delete_index(vectorBucketName=BUCKET_NAME, indexName=INDEX_NAME)
s3vectors.delete_vector_bucket(vectorBucketName=BUCKET_NAME)
print("Cleaned up resources")
if __name__ == "__main__":
setup_index()
load_movies()
search("epic space adventure with heroes")
search("family friendly adventure")
search("mind-bending thriller", genre_filter="scifi")
# Uncomment to clean up:
# cleanup()
What’s Next
You’ve learned how to use S3 Vectors for serverless semantic search. From here, you could:
- Scale up: Load a larger dataset (S3 Vectors supports up to 2 billion vectors per index)
- Ship a full search app: Build an end-to-end semantic search pipeline on S3 Vectors and Bedrock in Building CFP Oracle
- Add two-stage reranking: Pair vector recall with cross-encoder scoring in Cross-Encoders and Reranking
- Add hybrid search: Combine sparse BM25 lexical recall, dense vectors, RRF fusion, and reranking in Two-Stage Hybrid Retrieval
- Build a RAG pipeline: Connect to Claude for question-answering over your documents
Further Reading:
Keep Exploring
Related Guides
Visual Semantic Search with CLIP and AWS
Build a production image search system using OpenAI's CLIP model, Amazon OpenSearch Serverless for vector storage, and Claude on Bedrock for image descriptions. Complete Python implementation with real AWS outputs.
Two-Stage Hybrid Retrieval: BM25, Dense Embeddings, RRF, and Cross-Encoder Reranking
Build and benchmark a two-stage hybrid search pipeline combining Okapi BM25, BGE bi-encoder embeddings, Reciprocal Rank Fusion (RRF), and MS MARCO cross-encoder reranking across exact-identifier and semantic-paraphrase slices.
Cross-Attention Fusion: Combining Text Embeddings with Structured Features
Concatenation is the default. Here's why cross-attention works better for combining text embeddings with tabular data, and how to implement it in PyTorch.
Discussion
to join the discussion.