Glue

Step 1 of 8: Name it

What to call this connection, and the deployment it belongs to.

Step 2 of 8: Name the region

The AWS region the catalog is in.

Enter the AWS region the catalog is in. Glue is regional, so connect each region separately.

Step 3 of 8: Create the read-only user

A read-only IAM user in your account, and its key.

Make one IAM user for Convalesce in your own account with only the policy below, and issue it an access key. The commands do all three.

The policy allows reading the catalog, its databases and its tables, listing jobs with their lineage, and querying the lake's tables through Athena. The only thing it writes is each query's result, to the location you name below.

Athena needs somewhere to put a query's result. Name a bucket and prefix you are happy for it to write to; the last command points the primary workgroup at it. Leave that command out if your workgroup already has a result location, and name that location here so the policy matches.

This policy lets Convalesce read rows as well as the shape of your tables. While it investigates a failure the agent may run small read-only queries to confirm a cause: capped, with personal columns masked, never stored, and sent to the AI model. You can switch that off for this connection on its Settings tab once it is connected, without changing the policy.

Jobs whose scripts read from a named Glue connection also need glue:GetConnection, and reading a job script for lineage needs s3:GetObject on that script's bucket only. Add them for those jobs; never s3:GetObject on *.

There is no network rule to add: Convalesce calls AWS's own API, which is public. The exception is a bucket policy, a key policy or an IAM condition that limits source addresses (aws:SourceIp): it has to allow 34.66.85.47, the address Convalesce connects from.

cat > convalesce-read.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "glue:GetDatabases", "glue:GetDatabase", "glue:GetTables", "glue:GetTable",
        "glue:GetPartitions", "glue:GetPartition"
      ],
      "Resource": [
        "arn:aws:glue:us-east-1:123456789012:catalog",
        "arn:aws:glue:us-east-1:123456789012:database/*",
        "arn:aws:glue:us-east-1:123456789012:table/*/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": ["glue:GetJobs", "glue:GetDataflowGraph"],
      "Resource": "*"
    },
    {
      "Sid": "QueryTheLakeThroughAthena",
      "Effect": "Allow",
      "Action": [
        "athena:StartQueryExecution", "athena:GetQueryExecution",
        "athena:GetQueryResults", "athena:StopQueryExecution", "athena:GetWorkGroup"
      ],
      "Resource": "arn:aws:athena:us-east-1:123456789012:workgroup/primary"
    },
    {
      "Sid": "ReadTheLakesFiles",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": ["arn:aws:s3:::my-lake", "arn:aws:s3:::my-lake/*"]
    },
    {
      "Sid": "WriteQueryResults",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject", "s3:PutObject", "s3:AbortMultipartUpload",
        "s3:ListBucket", "s3:GetBucketLocation"
      ],
      "Resource": ["arn:aws:s3:::my-lake", "arn:aws:s3:::my-lake/athena-results/*"]
    }
  ]
}
EOF
aws iam get-user --user-name convalesce-reader >/dev/null 2>&1 || aws iam create-user --user-name convalesce-reader
aws iam put-user-policy --user-name convalesce-reader --policy-name ConvalesceGlueRead \
  --policy-document file://convalesce-read.json
aws iam create-access-key --user-name convalesce-reader
aws athena update-work-group --region us-east-1 --work-group primary \
  --configuration-updates "ResultConfigurationUpdates={OutputLocation=s3://my-lake/athena-results/}"

Run it as an IAM admin in your own account, where the AWS CLI is signed in (set AWS_PROFILE first if you use a named profile). It is safe to run again: the user is made once, and each tool's policy goes on under its own name. The last command prints AccessKeyId and SecretAccessKey once: paste them straight into the next step, nowhere else. AWS allows two keys per user, so delete an old one before issuing a third. In the IAM console instead, paste the JSON between the EOF lines as an inline policy.

Step 4 of 8: Add the access key

An access key for the reading identity.

An access key for an IAM user in your account that has only the catalog policy above. Convalesce has no AWS identity of its own, so it reads only what that user is granted.

Step 5 of 8: Choose what is read (optional)

Optional: narrow it to some databases and tables.

Everything the credential can see is read unless you narrow it here. List the databases and tables you want, the ones to leave out, or both.

Step 6 of 8: Test the connection

Check Convalesce can reach it with what you entered.

The test runs on the same worker a real run would, with the recipe exactly as it will be saved, so it fails the way a run would.

Step 7 of 8: Choose how often

How often Convalesce reads it.

Step 8 of 8: Review and connect

Check everything, then save the connection.