AWS Bedrock LLM

AWS Bedrock offers a very simple deployment service of full state of the art un-quantized foundational LLMs for you to run on your AWS account. While these models and their data should be private to your account each model does have its own EULA and can vary from model to model.

Connecting to AWS Bedrock

️💡

Valid account setup required!

You must have a valid AWS Account (opens in a new tab) to use AWS Bedrock - this service is not free.

You must be an administrator or root user of the account to follow the below steps. Otherwise, consult with your account admin.

Obtain IAM credentials to use for Bedrock

Heads up! You can optionally use session tokens for your AWS account if you do not want to create a new IAM user. Both forms of authentication are supported for AnythingLLM's AWS Bedrock integration. Learn more about temporary session tokens on AWS → (opens in a new tab)

the following below is for using IAM user authentication. If you are using session tokens this section is not relevant.

🚫

Do not use your root AWS access ID and Key for Bedrock. You should always use the least privileged policy possible when using AWS keys in any third-party applications, not just AnythingLLM.

Ideally, you should create a new user with at least the following permission and policy below. The below policy does not limit access to any particular model, but is good for starting. It is up to you to modify the access level of whatever credentials will be used with AWS Bedrock depending on your use-case.

The below credentials allow a user to see models and stream responses as well as default synchronous response - which is used for @agents.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "bedrock:InvokeModel",
        "bedrock:ListCustomModels",
        "bedrock:GetPrompt",
        "bedrock:GetFoundationModel",
        "bedrock:InvokeModelWithResponseStream",
        "bedrock:GetCustomModel",
        "bedrock:ListFoundationModels"
      ],
      "Resource": "*"
    }
  ]
}

Once a user with the correct permissions is defined or found you will need their AWS ACCESS ID and AWS ACCESS SECRET for connecting.

Copy the desired model you wish to use.

️💡

Please only use a chat capable model for LLM selection. Incompatible model types will not produce chats in AnythingLLM.

All models on Bedrock require permission from the model owner for use on your AWS account. Access is usually granted very quickly. You can request model permission from a link like this (replace us-west-2 with you region) https://us-west-2.console.aws.amazon.com/bedrock/home?region=us-west-2#/modelaccess (opens in a new tab)

Below are where you can find model ids that you can use in AnythingLLM.

Now that you have set up and have AWS Bedrock ready for LLM chatting, you can input the credentials to AnythingLLM.

AWS Bedrock IAM Access ID

This is the ACCESS ID of the user who can invoke models on Bedrock

AWS Bedrock IAM Access Key

This is the ACCESS SECRET of the user who can invoke models on Bedrock

AWS Bedrock Session Token

This is the session token of the user who can invoke models on Bedrock. You must be using temporary session tokens to use this and have selected the Session Token form of authentication in the UI. The default form of authentication is using IAM user credentials.

AWS region

This is the region that your account is enabled to use with AWS Bedrock - this depends on your region preference and account settings. An example is us-west-2 Oregon. Yours may be elsewhere.

Model ID

This is the model id copied from the screenshot above. If you are using a custom model, your key will be available elsewhere in the Bedrock UI.

Model context window

This is the maximum amount of tokens that can exist in a single query for a model. This is a model specific parameter and will vary from model to model. Consult with AWS Bedrock's UI or the model provider documentation for what this limit may be.

Azure OpenAI Cohere