Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 18 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,9 @@ Play Framework APIs for the Sunbird Knowledge Platform. Each service exposes RES
- [Step 2 — Start infrastructure](#step-2--start-infrastructure)
- [Step 3 — Initialize YugabyteDB keyspaces](#step-3--initialize-yugabytedb-keyspaces)
- [Step 4 — Initialize Elasticsearch indices](#step-4--initialize-elasticsearch-indices)
- [Step 5 — Build the project](#step-5--build-the-project)
- [Step 6 — Run a service](#step-6--run-a-service)
- [Step 5 — Populate Seed data](#step-5--seed-data)
- [Step 6 — Build the project](#step-6--build-the-project)
- [Step 7 — Run a service](#step-7--run-a-service)
4. [Redis (optional)](#redis-optional)
5. [Cloud Storage Configuration](#cloud-storage-configuration)
6. [CI/CD — GitHub Actions](#cicd--github-actions)
Expand Down Expand Up @@ -109,7 +110,19 @@ This downloads index and mapping definitions from [sunbird-devops](https://githu

You only need to run this once. Run it again after `docker compose down -v` (which deletes volumes).

### Step 5 — Build the project
### Step 5 — Populate seed data

Still inside the `docker/` directory, Populate the database with seed data script:

```shell
# Populate data into the default 'dev' environment
./seed-data.sh

# Populate data into a custom environment (e.g., 'sb')
./seed-data.sh sb
```

### Step 6 — Build the project

Go back to the repository root and build:

Expand All @@ -127,7 +140,7 @@ mvn clean install -DskipTests -Pgcp # Google Cloud Storage
mvn clean install -DskipTests -Poci # Oracle Cloud Infrastructure
```

### Step 6 — Run a service
### Step 7 — Run a service

> **Required:** Set [cloud storage environment variables](#cloud-storage-configuration) before starting any service. The `StorageModule` initializes eagerly on startup and the service will fail if the variables are empty. If you don't have real credentials, set placeholder values — storage will only fail when you actually upload/download content:
> ```shell
Expand Down Expand Up @@ -305,4 +318,4 @@ The project uses **GitHub Actions** for CI/CD. Workflows are defined in `.github
|--------|---------|
| `AWS_ACCESS_KEY_ID` | AWS access key ID |
| `AWS_SECRET_ACCESS_KEY` | AWS secret access key |
| `AWS_REGION` | `us-east-1` |
| `AWS_REGION` | `us-east-1` |
19 changes: 19 additions & 0 deletions docker/data-seed/category_store.category_definition_data.csv

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions docker/data-seed/dialcode_store.system_config.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
prop_key,prop_value
dialcode_max_index,1
91 changes: 91 additions & 0 deletions docker/data-seed/graph_snapshot.json

Large diffs are not rendered by default.

3 changes: 3 additions & 0 deletions docker/data-seed/hierarchy_store.framework_hierarchy.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
identifier,hierarchy
TPD,"{\"identifier\":\"TPD\",\"code\":\"TPD\",\"channel\":\"sunbird\",\"description\":\"Sunbird TPD framework\",\"languageCode\":[],\"type\":\"TPD\",\"createdOn\":\"2026-05-08T11:31:42.210+0000\",\"versionKey\":\"114567341278912512190\",\"objectType\":\"Framework\",\"systemDefault\":\"Yes\",\"name\":\"TPD\",\"lastUpdatedOn\":\"2026-05-08T11:31:42.210+0000\",\"categories\":[{\"identifier\":\"tpd_board\",\"code\":\"board\",\"terms\":[{\"identifier\":\"tpd_board_mscert\",\"code\":\"mscert\",\"name\":\"State (Maharashtra)\",\"index\":1,\"category\":\"board\",\"status\":\"Live\"},{\"identifier\":\"tpd_board_cbse\",\"code\":\"cbse\",\"name\":\"CBSE\",\"index\":2,\"category\":\"board\",\"status\":\"Live\"},{\"identifier\":\"tpd_board_ka\",\"code\":\"ka\",\"name\":\"State (Karnataka)\",\"index\":3,\"category\":\"board\",\"status\":\"Live\"},{\"identifier\":\"tpd_board_statetelangana\",\"code\":\"statetelangana\",\"name\":\"State (Telangana)\",\"index\":4,\"category\":\"board\",\"status\":\"Live\"}],\"name\":\"Board\",\"index\":1,\"status\":\"Live\"},{\"identifier\":\"tpd_medium\",\"code\":\"medium\",\"terms\":[{\"identifier\":\"tpd_medium_english\",\"code\":\"english\",\"name\":\"English\",\"index\":1,\"category\":\"medium\",\"status\":\"Live\"},{\"identifier\":\"tpd_medium_tamil\",\"code\":\"tamil\",\"name\":\"Tamil\",\"index\":2,\"category\":\"medium\",\"status\":\"Live\"},{\"identifier\":\"tpd_medium_telugu\",\"code\":\"telugu\",\"name\":\"Telugu\",\"index\":3,\"category\":\"medium\",\"status\":\"Live\"},{\"identifier\":\"tpd_medium_hindi\",\"code\":\"hindi\",\"name\":\"Hindi\",\"index\":4,\"category\":\"medium\",\"status\":\"Live\"}],\"name\":\"Medium\",\"index\":2,\"status\":\"Live\"},{\"identifier\":\"tpd_gradelevel\",\"code\":\"gradeLevel\",\"terms\":[{\"identifier\":\"tpd_gradelevel_class1\",\"code\":\"class1\",\"name\":\"Class 1\",\"index\":1,\"category\":\"gradeLevel\",\"status\":\"Live\"},{\"identifier\":\"tpd_gradelevel_class2\",\"code\":\"class2\",\"name\":\"Class 2\",\"index\":2,\"category\":\"gradeLevel\",\"status\":\"Live\"},{\"identifier\":\"tpd_gradelevel_class3\",\"code\":\"class3\",\"name\":\"Class 3\",\"index\":3,\"category\":\"gradeLevel\",\"status\":\"Live\"},{\"identifier\":\"tpd_gradelevel_class4\",\"code\":\"class4\",\"name\":\"Class 4\",\"index\":4,\"category\":\"gradeLevel\",\"status\":\"Live\"}],\"name\":\"Grade Level\",\"index\":3,\"status\":\"Live\"},{\"identifier\":\"tpd_subject\",\"code\":\"subject\",\"terms\":[{\"identifier\":\"tpd_subject_english\",\"code\":\"english\",\"name\":\"English\",\"index\":1,\"category\":\"subject\",\"status\":\"Live\"},{\"identifier\":\"tpd_subject_hindi\",\"code\":\"hindi\",\"name\":\"Hindi\",\"index\":2,\"category\":\"subject\",\"status\":\"Live\"},{\"identifier\":\"tpd_subject_tamil\",\"code\":\"tamil\",\"name\":\"Tamil\",\"index\":3,\"category\":\"subject\",\"status\":\"Live\"},{\"identifier\":\"tpd_subject_telugu\",\"code\":\"telugu\",\"name\":\"Telugu\",\"index\":4,\"category\":\"subject\",\"status\":\"Live\"}],\"name\":\"Subject\",\"index\":4,\"status\":\"Live\"}],\"status\":\"Live\"}"
NCF,"{\"identifier\":\"NCF\",\"code\":\"NCF\",\"channel\":\"sunbird\",\"description\":\"Sunbird k-12 framework\",\"languageCode\":[],\"type\":\"K-12\",\"createdOn\":\"2026-05-08T11:31:41.719+0000\",\"versionKey\":\"114567341274882048188\",\"objectType\":\"Framework\",\"systemDefault\":\"Yes\",\"name\":\"CBSE\",\"lastUpdatedOn\":\"2026-05-08T11:31:41.719+0000\",\"categories\":[{\"identifier\":\"ncf_board\",\"code\":\"board\",\"terms\":[{\"identifier\":\"ncf_board_mscert\",\"code\":\"mscert\",\"name\":\"State (Maharashtra)\",\"index\":1,\"category\":\"board\",\"status\":\"Live\"},{\"associations\":[{\"identifier\":\"ncf_medium_english\",\"code\":\"english\",\"name\":\"English\",\"index\":1,\"category\":\"medium\",\"status\":\"Live\"},{\"identifier\":\"ncf_medium_hindi\",\"code\":\"hindi\",\"name\":\"Hindi\",\"index\":1,\"category\":\"medium\",\"status\":\"Live\"}],\"identifier\":\"ncf_board_cbse\",\"code\":\"cbse\",\"name\":\"CBSE\",\"index\":2,\"category\":\"board\",\"status\":\"Live\"},{\"identifier\":\"ncf_board_ka\",\"code\":\"ka\",\"name\":\"State (Karnataka)\",\"index\":3,\"category\":\"board\",\"status\":\"Live\"},{\"identifier\":\"ncf_board_statetelangana\",\"code\":\"statetelangana\",\"name\":\"State (Telangana)\",\"index\":4,\"category\":\"board\",\"status\":\"Live\"}],\"name\":\"Board\",\"index\":1,\"status\":\"Live\"},{\"identifier\":\"ncf_medium\",\"code\":\"medium\",\"terms\":[{\"associations\":[{\"identifier\":\"ncf_gradelevel_class3\",\"code\":\"class3\",\"name\":\"Class 3\",\"index\":1,\"category\":\"gradeLevel\",\"status\":\"Live\"},{\"identifier\":\"ncf_gradelevel_class1\",\"code\":\"class1\",\"name\":\"Class 1\",\"index\":1,\"category\":\"gradeLevel\",\"status\":\"Live\"},{\"identifier\":\"ncf_gradelevel_class2\",\"code\":\"class2\",\"name\":\"Class 2\",\"index\":1,\"category\":\"gradeLevel\",\"status\":\"Live\"}],\"identifier\":\"ncf_medium_english\",\"code\":\"english\",\"name\":\"English\",\"index\":1,\"category\":\"medium\",\"status\":\"Live\"},{\"identifier\":\"ncf_medium_tamil\",\"code\":\"tamil\",\"name\":\"Tamil\",\"index\":2,\"category\":\"medium\",\"status\":\"Live\"},{\"identifier\":\"ncf_medium_telugu\",\"code\":\"telugu\",\"name\":\"Telugu\",\"index\":3,\"category\":\"medium\",\"status\":\"Live\"},{\"identifier\":\"ncf_medium_hindi\",\"code\":\"hindi\",\"name\":\"Hindi\",\"index\":4,\"category\":\"medium\",\"status\":\"Live\"}],\"name\":\"Medium\",\"index\":2,\"status\":\"Live\"},{\"identifier\":\"ncf_gradelevel\",\"code\":\"gradeLevel\",\"terms\":[{\"associations\":[{\"identifier\":\"ncf_subject_hindi\",\"code\":\"hindi\",\"name\":\"Hindi\",\"index\":1,\"category\":\"subject\",\"status\":\"Live\"},{\"identifier\":\"ncf_subject_english\",\"code\":\"english\",\"name\":\"English\",\"index\":1,\"category\":\"subject\",\"status\":\"Live\"}],\"identifier\":\"ncf_gradelevel_class1\",\"code\":\"class1\",\"name\":\"Class 1\",\"index\":1,\"category\":\"gradeLevel\",\"status\":\"Live\"},{\"identifier\":\"ncf_gradelevel_class2\",\"code\":\"class2\",\"name\":\"Class 2\",\"index\":2,\"category\":\"gradeLevel\",\"status\":\"Live\"},{\"identifier\":\"ncf_gradelevel_class3\",\"code\":\"class3\",\"name\":\"Class 3\",\"index\":3,\"category\":\"gradeLevel\",\"status\":\"Live\"},{\"identifier\":\"ncf_gradelevel_class4\",\"code\":\"class4\",\"name\":\"Class 4\",\"index\":4,\"category\":\"gradeLevel\",\"status\":\"Live\"}],\"name\":\"Grade Level\",\"index\":3,\"status\":\"Live\"},{\"identifier\":\"ncf_subject\",\"code\":\"subject\",\"terms\":[{\"identifier\":\"ncf_subject_english\",\"code\":\"english\",\"name\":\"English\",\"index\":1,\"category\":\"subject\",\"status\":\"Live\"},{\"identifier\":\"ncf_subject_hindi\",\"code\":\"hindi\",\"name\":\"Hindi\",\"index\":2,\"category\":\"subject\",\"status\":\"Live\"},{\"identifier\":\"ncf_subject_tamil\",\"code\":\"tamil\",\"name\":\"Tamil\",\"index\":3,\"category\":\"subject\",\"status\":\"Live\"},{\"identifier\":\"ncf_subject_telugu\",\"code\":\"telugu\",\"name\":\"Telugu\",\"index\":4,\"category\":\"subject\",\"status\":\"Live\"}],\"name\":\"Subject\",\"index\":4,\"status\":\"Live\"}],\"status\":\"Live\"}"
15 changes: 15 additions & 0 deletions docker/data-seed/import_graph.groovy
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
import org.apache.tinkerpop.gremlin.driver.remote.DriverRemoteConnection
import static org.apache.tinkerpop.gremlin.process.traversal.AnonymousTraversalSource.traversal

// Define connection
conn = DriverRemoteConnection.using('/opt/bitnami/janusgraph/conf/remote.yaml')
g = traversal().withRemote(conn)

try {
// Execute the drop and read
g.V().drop().iterate()
g.io("/tmp/graph_snapshot.json").read().iterate()
} finally {
// Ensure connection is closed
conn.close()
}
57 changes: 57 additions & 0 deletions docker/seed-data.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
#!/bin/bash
set -e

# Path to the data-seed directory
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
DATA_SEED_DIR="${SCRIPT_DIR}/data-seed"

# Environment prefix
ENV=${1:-dev}

# Force Reset option
FORCE_RESET=${FORCE_RESET:-false}

echo "Starting data restoration for environment: ${ENV} from ${DATA_SEED_DIR}..."

# Find all CSV files in the data-seed directory
for file in "${DATA_SEED_DIR}"/*.csv; do
[ -e "$file" ] || continue

filename=$(basename "$file")

# Split filename into parts (keyspace.table.csv)
IFS='.' read -r keyspace table ext <<< "$filename"

target_table="${ENV}_${keyspace}.${table}"

echo "Restoring table: ${target_table}"

# Copy file to container
docker cp "$file" yugabyte:/tmp/"$filename"

# Optional: Truncate
if [ "$FORCE_RESET" = "true" ]; then
echo " Truncating ${target_table}"
MSYS_NO_PATHCONV=1 docker exec yugabyte /home/yugabyte/bin/ycqlsh -e "TRUNCATE $target_table;"
fi

# Execute COPY FROM
MSYS_NO_PATHCONV=1 docker exec yugabyte /home/yugabyte/bin/ycqlsh -e "COPY $target_table FROM '/tmp/$filename' WITH HEADER=TRUE;"
echo " Restored ${target_table}"
done

# JanusGraph Import
GRAPH_SNAPSHOT="${DATA_SEED_DIR}/graph_snapshot.json"
if [ -f "$GRAPH_SNAPSHOT" ]; then
echo "Restoring JanusGraph data..."
docker cp "$GRAPH_SNAPSHOT" janusgraph:/tmp/graph_snapshot.json
docker cp "${DATA_SEED_DIR}/import_graph.groovy" janusgraph:/tmp/import_graph.groovy

# Execute Gremlin import
docker exec -u 0 janusgraph bash -c 'HADOOP_GREMLIN_LIBS="" /opt/bitnami/janusgraph/bin/gremlin.sh -e /tmp/import_graph.groovy'
echo "Restored JanusGraph data."
else
echo "No graph snapshot found, skipping JanusGraph restoration."
fi

echo "Data restoration complete."
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import org.apache.commons.lang3.StringUtils
import org.sunbird.cloudstore.StorageService
import org.sunbird.common.Platform
import org.sunbird.mimetype.mgr.MimeTypeManager
import org.sunbird.mimetype.mgr.impl.{ApkMimeTypeMgrImpl, AssetMimeTypeMgrImpl, CollectionMimeTypeMgrImpl, DefaultMimeTypeMgrImpl, DocumentMimeTypeMgrImpl, EcmlMimeTypeMgrImpl, H5PMimeTypeMgrImpl, HtmlMimeTypeMgrImpl, PluginMimeTypeMgrImpl, YouTubeMimeTypeMgrImpl}
import org.sunbird.mimetype.mgr.impl.{ApkMimeTypeMgrImpl, AssetMimeTypeMgrImpl, CollectionMimeTypeMgrImpl, DefaultMimeTypeMgrImpl, DocumentMimeTypeMgrImpl, EcmlMimeTypeMgrImpl, H5PMimeTypeMgrImpl, HtmlMimeTypeMgrImpl, PluginMimeTypeMgrImpl, ScormMimeTypeMgrImpl, YouTubeMimeTypeMgrImpl}

object MimeTypeManagerFactory {

Expand All @@ -23,6 +23,7 @@ object MimeTypeManagerFactory {
"application/vnd.ekstep.content-collection" -> new CollectionMimeTypeMgrImpl,
"application/vnd.ekstep.plugin-archive" -> new PluginMimeTypeMgrImpl,
"application/vnd.ekstep.h5p-archive" -> new H5PMimeTypeMgrImpl,
"application/vnd.ekstep.scorm-archive" -> new ScormMimeTypeMgrImpl,
"application/vnd.android.package-archive" -> new ApkMimeTypeMgrImpl
)
if(ONLINE_MIMETYPES.contains(mimeType))
Expand All @@ -35,4 +36,4 @@ object MimeTypeManagerFactory {
else defaultMimeTypeMgrImpl
}
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ class BaseMimeTypeManager(implicit ss: StorageService) {
private val CONTENT_FOLDER = "cloud_storage.content.folder"
private val ARTIFACT_FOLDER = "cloud_storage.artifact.folder"
private val validator = new UrlValidator()
protected val extractableMimeTypes = List("application/vnd.ekstep.ecml-archive", "application/vnd.ekstep.html-archive", "application/vnd.ekstep.plugin-archive", "application/vnd.ekstep.h5p-archive")
protected val extractableMimeTypes = List("application/vnd.ekstep.ecml-archive", "application/vnd.ekstep.html-archive", "application/vnd.ekstep.plugin-archive", "application/vnd.ekstep.h5p-archive", "application/vnd.ekstep.scorm-archive")
protected val extractablePackageExtensions = List(".zip", ".h5p", ".epub")
private val H5P_MIMETYPE: String = "application/vnd.ekstep.h5p-archive"
private val H5P_LIBRARY_PATH: String = Platform.config.getString("content.h5p.library.path")
Expand Down Expand Up @@ -140,17 +140,18 @@ class BaseMimeTypeManager(implicit ss: StorageService) {
val resolvedBaseDir = baseDir.toRealPath()
val zipFile = new ZipFile(file)
try {
for (entry <- zipFile.entries().asScala) {
val path = resolvedBaseDir.resolve(entry.getName).normalize()
if (!path.startsWith(resolvedBaseDir))
throw new ClientException("ERR_INVALID_ZIP_ENTRY",
"Zip entry attempts path traversal: " + entry.getName)
if (entry.isDirectory) Files.createDirectories(path)
else {
Files.createDirectories(path.getParent)
Files.copy(zipFile.getInputStream(entry), path)
}
for (entry <- zipFile.entries().asScala) {
val path = resolvedBaseDir.resolve(entry.getName).normalize()
if (!path.startsWith(resolvedBaseDir))
throw new ClientException("ERR_INVALID_ZIP_ENTRY",
"Zip entry attempts path traversal: " + entry.getName)

if (entry.isDirectory) Files.createDirectories(path)
else {
Files.createDirectories(path.getParent)
Files.copy(zipFile.getInputStream(entry), path)
}
}
} finally {
zipFile.close()
}
Expand Down Expand Up @@ -212,6 +213,7 @@ class BaseMimeTypeManager(implicit ss: StorageService) {
case "application/vnd.ekstep.html-archive" => baseFolder + File.separator + "html" + File.separator + objectId + DASH + pathSuffix
case "application/vnd.ekstep.h5p-archive" => baseFolder + File.separator + "h5p" + File.separator + objectId + DASH + pathSuffix
case "application/vnd.ekstep.plugin-archive" => CONTENT_PLUGINS + File.separator + objectId + DASH + pathSuffix
case "application/vnd.ekstep.scorm-archive" => baseFolder + File.separator + "scorm" + File.separator + objectId + DASH + pathSuffix
case _ => ""
}
}
Expand Down Expand Up @@ -305,4 +307,3 @@ class BaseMimeTypeManager(implicit ss: StorageService) {
}

}

Loading