sciDX API is a RESTful API built with FastAPI that plays a crucial role in the sciDX stack. This project is designed to facilitate the management and processing of data streams from various sources, integrating services like CKAN and Keycloak for authentication, authorization, and data handling.
The sciDX API is a key component within the sciDX ecosystem, a platform designed to manage and process scientific data efficiently and securely. The API enables:
- Data Stream Management: Processes and manages real-time data using Kafka.
- CKAN Integration: Facilitates the integration and manipulation of datasets in CKAN.
- Authentication and Authorization: Implements access control using Keycloak.
- URL Handling and Data Streaming: Manages data ingestion, updating, and processing through URLs and streaming services.
The API enables the publishing and consumption of messages via Kafka, facilitating real-time data processing in a distributed environment.
sciDX API integrates with CKAN to create, update, and delete datasets, ensuring that scientific data is accessible and efficiently managed.
Using Keycloak, the API manages user authentication and authorization, providing robust access control.
The API supports data ingestion through URLs, allowing for dynamic management and updating of external data sources.
The sciDX API can be configured for use as a Point of Presence (POP) within the NDP project. When the POP configuration is enabled, specific API routes, such as those related to stream management, are hidden to tailor the API's functionality for POP deployments.
For detailed instructions on installing, configuring, and accessing the sciDX API, please see the Installation Guide.
Contributions are welcome. Please review our Contributing Guide for more details.
All tests are designed to run with pytest. Before submitting new contributions, please ensure that all existing tests pass successfully. Additionally, for any new features or functionalities introduced, create appropriate tests without modifying the existing ones. This helps maintain the integrity and reliability of the API.
This project is licensed under the MIT License.