Stop Guessing Which Model Is Better: Amazon Bedrock Model Evaluation Hands-On

"Which model should we use?" is the most common question in every Bedrock project — and most teams answer it by eyeballing a few responses. That doesn't scale, it isn't reproducible, and it silently expires every time a new model version ships.

In this hands-on, we'll answer the question with data: Amazon Bedrock Model Evaluation, run in two modes — automatic metrics scored against reference answers, and LLM-as-a-Judge, where a stronger model grades each response. We'll build the evaluation dataset, run the jobs, and crunch the result files down to comparable numbers with jq and awk.

Prefer video? This entire hands-on is also on YouTube:

The two evaluation modes

Automatic evaluation runs your dataset through the target model and scores each response against your referenceResponse with built-in metrics — accuracy-style similarity scores, robustness, toxicity. Fast, cheap, objective, but only as good as your reference answers.

LLM-as-a-Judge has a judge model read each prompt/response pair and grade qualities like correctness, completeness, and helpfulness. It catches what string-similarity metrics can't — a response can be worded completely differently from the reference and still be right — at the cost of running a second, stronger model.

Run both and you get two independent views of the same model, the same defense-in-depth idea applied to quality instead of security.

Note on model IDs: Bedrock models are updated frequently — pick current models when you create the evaluation jobs, not whatever a months-old article names.

Step 1: Environment and bucket

All of this runs in CloudShell:

export REGION=us-east-1
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export EVAL_BUCKET="bedrock-eval-$ACCOUNT_ID"

aws s3 mb "s3://$EVAL_BUCKET" --region $REGION

Step 2: The evaluation dataset

The dataset is JSONL — one JSON object per line, each with a prompt and a referenceResponse (the answer you consider correct). Ten AWS-basics Q&A pairs:

cat > /tmp/eval-dataset.jsonl << 'EOF'
{"prompt":"Explain the main use of AWS Lambda in one sentence","referenceResponse":"A compute service that runs code serverlessly in response to events"}
{"prompt":"What kind of service is Amazon S3?","referenceResponse":"A highly available and highly durable object storage service"}
{"prompt":"What kind of database is Amazon DynamoDB?","referenceResponse":"A fully managed NoSQL key-value database"}
{"prompt":"What is a CloudWatch alarm?","referenceResponse":"A mechanism that sends notifications or triggers automated actions when a metric crosses a threshold"}
{"prompt":"What is an IAM role?","referenceResponse":"A mechanism for granting temporary permissions to AWS resources and users"}
{"prompt":"What is Amazon VPC?","referenceResponse":"A service for building a logically isolated virtual network on AWS"}
{"prompt":"What are the characteristics of Amazon RDS?","referenceResponse":"A service that runs relational databases in a fully managed way"}
{"prompt":"What is the main function of CloudFront?","referenceResponse":"A content delivery network (CDN) that uses edge locations"}
{"prompt":"What kind of service is Amazon SQS?","referenceResponse":"A managed message queuing service"}
{"prompt":"What is Amazon Bedrock?","referenceResponse":"A service that provides foundation models through a serverless, unified API"}
EOF

aws s3 cp /tmp/eval-dataset.jsonl "s3://$EVAL_BUCKET/input/dataset.jsonl"

Ten pairs is a hands-on size. The mechanics are identical at 500 — for a real project, this file is the asset worth investing in: it becomes your regression test for every future model release.

Step 3: IAM role for the evaluation jobs

Bedrock runs the evaluation on your behalf, so it needs a role it can assume, with read/write on the bucket and permission to invoke models:

cat > /tmp/eval-trust.json << 'EOF'
{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"bedrock.amazonaws.com"},"Action":"sts:AssumeRole"}]}
EOF

EVAL_ROLE_ARN=$(aws iam create-role \
  --role-name BedrockEvaluationRole \
  --assume-role-policy-document file:///tmp/eval-trust.json \
  --query 'Role.Arn' --output text)

cat > /tmp/eval-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {"Effect":"Allow","Action":["s3:GetObject","s3:ListBucket","s3:PutObject"],
     "Resource":["arn:aws:s3:::$EVAL_BUCKET","arn:aws:s3:::$EVAL_BUCKET/*"]},
    {"Effect":"Allow","Action":["bedrock:InvokeModel"],
     "Resource":["arn:aws:bedrock:*::foundation-model/*","arn:aws:bedrock:*:*:inference-profile/*"]}
  ]
}
EOF

aws iam put-role-policy --role-name BedrockEvaluationRole \
  --policy-name inline --policy-document file:///tmp/eval-policy.json

Note the trust policy: the principal is bedrock.amazonaws.com — this role is for the service, not for you or a Lambda.

Step 4: Create the two evaluation jobs (console)

In the Bedrock console (Inference and Assessment → Evaluations), create two jobs against the same dataset — the video above walks through every screen:

1. Automatic evaluation — pick the model to evaluate, task type Q&A, your dataset at s3://$EVAL_BUCKET/input/dataset.jsonl, the BedrockEvaluationRole, and an output path under s3://$EVAL_BUCKET/output/auto/
2. LLM-as-a-Judge — same dataset and role, pick the model to evaluate and a judge model (use a stronger model than the one being judged), output under s3://$EVAL_BUCKET/output/judge/

Both jobs run asynchronously — expect several minutes to tens of minutes depending on dataset size.

Step 5: Pull the results and make them readable

When the jobs complete, sync everything down and find the result files:

aws s3 sync "s3://$EVAL_BUCKET/output/" /tmp/eval-out/
find /tmp/eval-out/ -type f

# List the main output files
find /tmp/eval-out/ -name "*_output.jsonl" -not -name "*.out"

The output is deeply nested JSONL — one record per prompt, each carrying a scores array. Aggregate the automatic metrics into per-metric count / mean / max:

for f in /tmp/eval-out/auto/*/*/models/*/taskTypes/*/datasets/*/*_output.jsonl; do
  cat "$f" | jq -r '.automatedEvaluationResult.scores[] | "\(.metricName)\t\(.result)"'
done | awk -F'\t' '
  {sum[$1]+=$2; cnt[$1]++; if($2>max[$1]) max[$1]=$2}
  END {for (m in sum) printf "%s\tcount %d\tmean %.4f\tmax %.4f\n", m, cnt[m], sum[m]/cnt[m], max[m]}
'

And the judge's scores, averaged per metric:

JUDGE=$(find /tmp/eval-out/judge -name "*_output.jsonl" -not -name "*.out" | head -1)

cat "$JUDGE" | jq -r '.automatedEvaluationResult.scores[] | "\(.metricName)\t\(.result)"' | \
  awk '{sum[$1]+=$2; cnt[$1]++} END {for (m in sum) printf "%s\tmean %.2f\n", m, sum[m]/cnt[m]}'

Two numbers-reading tips:

• Don't compare across metric families. An automatic similarity score and a judge's correctness grade live on different scales; compare models within the same metric, not metrics with each other.
• The mean hides the failures. The per-record JSONL is right there — before trusting an average, look at the worst-scoring records and read what the model actually said. Ten minutes of reading beats any single number.

Cleanup

# S3
aws s3 rm "s3://$EVAL_BUCKET" --recursive
aws s3 rb "s3://$EVAL_BUCKET"

# IAM
aws iam delete-role-policy --role-name BedrockEvaluationRole --policy-name inline
aws iam delete-role --role-name BedrockEvaluationRole

Wrapping up

Model selection without evaluation is a vibe, and vibes don't survive the pace at which Bedrock ships new models. A ten-line JSONL file, one IAM role, and two evaluation jobs give you a repeatable benchmark you can rerun against every new release — and the same dataset doubles as a regression test when you change prompts, parameters, or routing. If you've been choosing models by eyeballing outputs, this is the upgrade.

About the author

Maruchin Tech — 12x AWS Certified | Cloud & AI for manufacturing and supply chain (AWS / Google Cloud / Azure) | Udemy instructor (100K+ students)

🎥 Video version of this hands-on:
https://youtu.be/jbrFKA34hWc

📚 Full course — AWS Certified Generative AI Developer Professional (AIP-C01) Exam Prep:
https://www.udemy.com/course/aws-certified-generative-ai-developer-professional-exam-prep/

👨‍🏫 All my courses:
(Eng) https://www.udemy.com/user/maruchin-tech-2/
(Jpn) https://www.udemy.com/user/shan-wang-wan-jun-2/

🎫 Monthly discount coupons:
https://www.youtube.com/@MaruchinTech-cloud/posts