How should a budget be implemented in AWS?

[dirkpetersen]

Please see https://github.com/dirkpetersen/our-chat?tab=readme-ov-file#Budgeting

[dirkpetersen]

ChatGPT likes this solution, not sure why DynamoDB is needed? Because of the lag of CE ?

import boto3
from datetime import datetime

# Initialize AWS clients
iam = boto3.client('iam')
dynamodb = boto3.resource('dynamodb')
ce = boto3.client('ce')

# Constants
USER_NAME = 'librechat'
POLICY_ARN = 'arn:aws:iam::aws:policy/AmazonBedrockFullAccess'
TABLE_NAME = 'BudgetTracker'

def get_monthly_spend():
    """Query AWS Cost Explorer for current month's spend."""
    start_date = datetime.now().replace(day=1).strftime('%Y-%m-%d')
    end_date = datetime.now().strftime('%Y-%m-%d')
    
    response = ce.get_cost_and_usage(
        TimePeriod={'Start': start_date, 'End': end_date},
        Granularity='MONTHLY',
        Metrics=['UnblendedCost']
    )
    
    spend_amount = float(response['ResultsByTime'][0]['Total']['UnblendedCost']['Amount'])
    return spend_amount

def check_and_update_access(accumulated_budget, total_spend):
    """Attach or detach Bedrock access based on budget constraints."""
    if total_spend > accumulated_budget:
        # Disable access if over budget
        iam.detach_user_policy(UserName=USER_NAME, PolicyArn=POLICY_ARN)
        return f'Access disabled for user {USER_NAME}.'
    else:
        # Enable access if within budget
        iam.attach_user_policy(UserName=USER_NAME, PolicyArn=POLICY_ARN)
        return f'Access enabled for user {USER_NAME}.'

def lambda_handler(event, context):
    # Get today's date
    today = datetime.utcnow().strftime('%Y-%m-%d')

    # Get current accumulated budget from DynamoDB
    table = dynamodb.Table(TABLE_NAME)
    response = table.get_item(Key={'Date': today})
    accumulated_budget = response.get('Item', {}).get('AccumulatedBudget', 32)
    
    # Get the current monthly spend
    total_spend = get_monthly_spend()

    # Check budget and update access
    result_message = check_and_update_access(accumulated_budget, total_spend)

    return {
        'statusCode': 200,
        'body': result_message
    }

[ldavis9000aws]

So what does your user base actually need access to within AWS Bedrock? Do you know how they plan to use the models, what problems they're trying to solve, etc.? That information will give you more insight into which LLM's they probably need access to & help you determine if they really need access to the more expensive LLM's (like Antrhopic Haiku or Opus vs just using Sonnet 3.5). I would take a page out of the AWS Free Tier approach and initially only give folks access to the most common LLM's that will be used based on their use cases. Also, perhaps limit the # of people who can use OurChat to get an idea of the costs you may encounter if you open it up to even more employees.

Other cost management ideas:

• Look at any caching solutions that can reduce the number of API requests to Bedrock
• Consider tagging different IAM service accounts (or AWS groups via SSO / AD Federation services) and issuing IAM credentials for them....within LibreChat, this may mean that each user needs to enter in their details at the beginning of each session, but at least you'd have a way to track Amazon Bedrock useage.

It would also be great if we could create profiles or roles with assigned access to different models & features within Librechat. I'm honestly just typing off the top of my head at this point, but it seems like this is a problem best solved in the upstream in terms of controlling costs. The AWS bill does breakdown which Bedrock models were invoked and the costs associated with each. So that might help in terms of understanding which ones are being used the most.

My 2 cents. :)

[ldavis9000aws]

@dirkpetersen Maybe it's a good idea to open up a discussion section as well.

[dirkpetersen]

Thanks @ldavis9000aws, I agree that a more sophisticated authorization and role based access model in combination with budgeting and cost control would be desirable for LibreChat. Currently you have to stand up a different instance for that. Our approach is multi-phased. We have a potential of about 35000 users and current only 5 have access via AD security group. For the foreseeable future, each user will be approved individually, based on the merit of their use case reviewed by a committee. However, neither the users nor most committee members are currently able to give an answer which model is best for what. There is a significant amount of trial and error. I would assume that once the organization and the users have become more comfortable with accessing LLMs (which can produce a lot of faulty output) we might open this up more if not all users. I see this as a good option for occasional users. Power users will be better off with a monthly subscription and / or direct API access to Bedrock.