fix: added OpenSearch security calls under feature flag

[lucaseduoli]

This pull request introduces a new set of OpenSearch security configurations specifically for cloud deployments, improves the application of Document-Level Security (DLS), and adds a comprehensive test script for validating ACL enforcement. It also updates Dockerfiles to include the new security configuration files and makes minor improvements to the OpenSearch index initialization process.

Cloud Security Configuration:

• Adds a new cloud_securityconfig/ directory with OpenSearch security configuration files for cloud environments, including roles.yml, roles_mapping.yml, config.yml, internal_users.yml, and others. These files define custom roles (such as openrag_user_role), role mappings, authentication domains (OIDC and internal), and DLS rules for fine-grained access control. [1] [2] [3] [4] [5] [6] [7]

• Updates the main securityconfig/roles.yml to include additional permissions and index patterns, ensuring parity with cloud settings and better DLS support. [1] [2]

Dockerfile and Build Updates:

• Modifies both Dockerfile and Dockerfile.backend to copy the new cloud_securityconfig/ directory into the build context and runtime images, and ensures correct file ownership for both standard and cloud security configs. [1] [2] [3]

Testing and Validation:

• Adds a new script scripts/test.sh that performs end-to-end validation of OpenSearch ACLs and DLS. The script checks cluster health, verifies user roles, inspects index mappings, and tests document visibility for different users, both with and without application-level ACL filters.

Backend Improvements:

• Updates the OpenSearch index initialization logic in src/api/settings.py to use the correct function and pass the admin username when IBM authentication is enabled, improving clarity and correctness during onboarding. [1] [2]

[edwinjosechittilappilly]

Crititcal issues with Docker image

[edwinjosechittilappilly]

@lucaseduoli 2 critical updates:

1. copy security config folder to backend image.
2. Currently the security config is applied to the docuemts folder for indexs and other folders, but the user might use a different index name from env, just like we do for deployemnts.
   Hence we need to ensure to add this index name dynamically to that list.

[edwinjosechittilappilly]

example:

    - index_patterns: ["documents", "documents*", "knowledge_filters", "knowledge_filters*"]

in roles.yml and downstream api calls that uses this.

[edwinjosechittilappilly]

LGTM