Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 70 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ To build overlaybd from source code, the following dependencies are required:
* Libaio, libcurl, libnl3, glib2 and openssl runtime and development libraries.
* CentOS 7/Fedora: `sudo yum install libaio-devel libcurl-devel openssl-devel libnl3-devel libzstd-static e2fsprogs-devel`
* CentOS 8: `sudo yum install libaio-devel libcurl-devel openssl-devel libnl3-devel libzstd-devel e2fsprogs-devel`
* Debian/Ubuntu: `sudo apt install libcurl4-openssl-dev libssl-dev libaio-dev libnl-3-dev libnl-genl-3-dev libgflags-dev libzstd-dev libext2fs-dev`
* Debian/Ubuntu: `sudo apt install libcurl4-openssl-dev libssl-dev libaio-dev libnl-3-dev libnl-genl-3-dev libgflags-dev libzstd-dev libext2fs-dev pkg-config automake libtool # libgtest-dev // for test`
* Mariner/AzureLinux: `sudo yum install libaio-devel libcurl-devel openssl-devel libnl3-devel e2fsprogs-devel glibc-devel libzstd-devel binutils ca-certificates-microsoft build-essential`

#### Build
Expand Down Expand Up @@ -169,7 +169,11 @@ Default configure file `overlaybd.json` is installed to `/etc/overlaybd/`.
"updateInterval": 60000000
},
"enableAudit": true,
"auditPath": "/var/log/overlaybd-audit.log"
"auditPath": "/var/log/overlaybd-audit.log",
"serviceConfig": {
"enable": false,
"address": "http://127.0.0.1:9862"
}
}
```

Expand Down Expand Up @@ -210,6 +214,8 @@ Default configure file `overlaybd.json` is installed to `/etc/overlaybd/`.
| certConfig.certFile | The path for SSL/TLS client certificate file |
| certConfig.keyFile | The path for SSL/TLS client key file |
| userAgent | customized userAgent to identify HTTP request. default value is package version like 'overlaybd/1.1.14-6c449832' |
| serviceConfig.enable | Enable live snapshot API service, `false` is default. |
| serviceConfig.address | API service listening address, default `http://127.0.0.1:9862`. |


> NOTE: `download` is the config for background downloading. After an overlaybd device is lauched, a background task will be running to fetch the whole blobs into local directories. After downloading, I/O requests are directed to local files. Unlike other options, download config is reloaded when a device launching.
Expand Down Expand Up @@ -381,6 +387,68 @@ At last, compression may be needed.
```
The zfile can be used as lower layer with online decompression.

### Live Snapshot

Overlaybd supports creating live snapshots without stopping the device. This feature allows you to capture the current state of a writable layer and stack a new writable layer on top.

#### Device ID

To use the live snapshot feature, you need to specify a device ID when creating the overlaybd device. The device ID is appended to the config path with a semicolon separator:

```bash
echo -n dev_config=overlaybd//root/config.v1.json;123 > /sys/kernel/config/target/core/user_1/vol1/control
```

#### Enable API Service

Add the following to your `overlaybd.json`:

```json
"serviceConfig": {
"enable": true,
"address": "http://127.0.0.1:9862"
}
```

#### Create Snapshot

Send an HTTP POST request to the `/snapshot` endpoint:

```bash
curl -X POST "http://127.0.0.1:9862/snapshot?dev_id=123&config=/path/to/new_config.json"
```

The response will be in JSON format:

```json
{
"success": true,
"message": "Snapshot created successfully"
}
```

#### New Config Format

The new config file should include the current upper layer as the last lower layer:

```json
{
"lowers": [
{
"file": "/opt/overlaybd/layer0"
},
{
"file": "/path/to/current_upper_data.lsmt"
}
],
"upper": {
"index": "/path/to/new_upper_index.lsmt",
"data": "/path/to/new_upper_data.lsmt"
}
}
```

**Note**: The new upper layer must be different from the old upper layer.

## Kernel module

Expand Down
1 change: 1 addition & 0 deletions src/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ add_library(overlaybd_image_lib
prefetch.cpp
tools/sha256file.cpp
tools/comm_func.cpp
api_server.cpp
)
target_include_directories(overlaybd_image_lib PUBLIC
${CURL_INCLUDE_DIRS}
Expand Down
142 changes: 142 additions & 0 deletions src/api_server.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
/*
Copyright The Overlaybd Authors

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

#include <photon/net/http/url.h>
#include <string_view>
#include "image_service.h"
#include "image_file.h"
#include "api_server.h"

int ApiHandler::handle_request(photon::net::http::Request& req,
photon::net::http::Response& resp,
std::string_view) {
auto target = req.target(); // string view, format: /snapshot?dev_id=${devID}&config=${config}
std::string_view query("");
auto pos = target.find('?');
if (pos != std::string_view::npos) {
query = target.substr(pos + 1);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use req.query() instead.

LOG_DEBUG("Snapshot query: `", query); // string view, format: dev_id=${devID}&config=${config}
parse_params(query);
auto dev_id = params["dev_id"];
auto config_path = params["config"];
LOG_DEBUG("dev_id: `, config: `", dev_id, config_path);

int code;
std::string msg;
ImageFile* img_file = nullptr;

if (dev_id.empty() || config_path.empty()) {
code = 400;
msg = std::string(R"delimiter({
"success": false,
"message": "Missing dev_id or config in snapshot request"
})delimiter");
goto EXIT;
}

img_file = imgservice->find_image_file(dev_id);
if (!img_file) {
code = 404;
msg = std::string(R"delimiter({
"success": false,
"message": "Image file not found"
})delimiter");
goto EXIT;
}

if (img_file->create_snapshot(config_path.c_str()) < 0) {
code = 500;
msg = std::string(R"delimiter({
"success": false,
"message": "Failed to create snapshot`"
})delimiter");
goto EXIT;
}

code = 200;
msg = std::string(R"delimiter({
"success": true,
"message": "Snapshot created successfully"
})delimiter");

EXIT:
resp.set_result(code);
resp.headers.content_length(msg.size());
resp.keep_alive(true);
auto ret_w = resp.write((void*)msg.c_str(), msg.size());
if (ret_w != (ssize_t)msg.size()) {
LOG_ERRNO_RETURN(0, -1, "send body failed, target: `, `", req.target(), VALUE(ret_w));
}
LOG_DEBUG("send body done");
return 0;
}

void ApiHandler::parse_params(std::string_view query) { // format: dev_id=${devID}&config=${config}...
if (query.empty())
return;

size_t start = 0;
while (start < query.length()) {
auto end = query.find('&', start);
if (end == std::string_view::npos) { // last one
end = query.length();
}

auto param = query.substr(start, end - start);
auto eq_pos = param.find('=');
if (eq_pos != std::string_view::npos) {
auto key = param.substr(0, eq_pos);
auto value = param.substr(eq_pos + 1);

// url decode
auto decoded_key = photon::net::http::url_unescape(key);
auto decoded_value = photon::net::http::url_unescape(value);
params[decoded_key] = decoded_value;
} else {
// key without value
auto key = photon::net::http::url_unescape(param);
params[key] = "";
}
start = end + 1;
}
}

ApiServer::ApiServer(const std::string &addr, ApiHandler* handler) {
photon::net::http::URL url(addr);
std::string host = url.host().data(); // the string pointed by data() doesn't end up with '\0'
auto pos = host.find(":");
if (pos != host.npos) {
host.resize(pos);
}
tcpserver = photon::net::new_tcp_socket_server();
tcpserver->setsockopt(SOL_SOCKET, SO_REUSEPORT, 1);
if(tcpserver->bind(url.port(), photon::net::IPAddr(host.c_str())) < 0)
LOG_ERRNO_RETURN(0, , "Failed to bind api server port `", url.port());
if(tcpserver->listen() < 0)
LOG_ERRNO_RETURN(0, , "Failed to listen api server port `", url.port());
httpserver = photon::net::http::new_http_server();
httpserver->add_handler(handler, false, "/snapshot");
tcpserver->set_handler(httpserver->get_connection_handler());
tcpserver->start_loop();
ready = true;
LOG_DEBUG("Api server listening on `:`, path: `", host, url.port(), "/snapshot");
}

ApiServer::~ApiServer() {
delete tcpserver;
delete httpserver;
}
45 changes: 45 additions & 0 deletions src/api_server.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
/*
Copyright The Overlaybd Authors

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

#pragma once

#include <map>
#include <string>
#include <photon/net/http/server.h>
#include <photon/net/socket.h>

class ImageService;

class ApiHandler : public photon::net::http::HTTPHandler {
public:
ImageService *imgservice;
std::map<std::string, std::string> params;

ApiHandler(ImageService *imgservice) : imgservice(imgservice) {}
int handle_request(photon::net::http::Request& req,
photon::net::http::Response& resp,
std::string_view) override;
void parse_params(std::string_view query);
};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's not necessary to expose ApiHandler in the header file. It's okay to just pass ImageService* to ApiServer.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you suggesting that I should define ApiHandler in the cpp file instead of the header, and then instantiate it inside the ApiServer constructor?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you suggesting that I should define ApiHandler in the cpp file instead of the header, and then instantiate it inside the ApiServer constructor?

Exactly.


struct ApiServer {
photon::net::ISocketServer* tcpserver = nullptr;
photon::net::http::HTTPServer* httpserver = nullptr;
bool ready = false;

ApiServer(const std::string &addr, ApiHandler* handler);
~ApiServer();
};
8 changes: 8 additions & 0 deletions src/config.h
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,12 @@ struct CertConfig : public ConfigUtils::Config {
APPCFG_PARA(keyFile, std::string, "");
};

struct ServiceConfig : public ConfigUtils::Config {
APPCFG_CLASS
APPCFG_PARA(enable, bool, false);
APPCFG_PARA(address, std::string, "http://127.0.0.1:9862");
};

struct GlobalConfig : public ConfigUtils::Config {
APPCFG_CLASS

Expand All @@ -155,6 +161,7 @@ struct GlobalConfig : public ConfigUtils::Config {
APPCFG_PARA(prefetchConfig, PrefetchConfig);
APPCFG_PARA(certConfig, CertConfig);
APPCFG_PARA(userAgent, std::string, OVERLAYBD_VERSION);
APPCFG_PARA(serviceConfig, ServiceConfig);
};

struct AuthConfig : public ConfigUtils::Config {
Expand All @@ -170,4 +177,5 @@ struct ImageAuthResponse : public ConfigUtils::Config {
APPCFG_PARA(data, AuthConfig);
};


} // namespace ImageConfigNS
6 changes: 5 additions & 1 deletion src/example_config/overlaybd.json
Original file line number Diff line number Diff line change
Expand Up @@ -36,5 +36,9 @@
},
"enableAudit": true,
"auditPath": "/var/log/overlaybd-audit.log",
"registryFsVersion": "v2"
"registryFsVersion": "v2",
"serviceConfig": {
"enable": false,
"address": "http://127.0.0.1:9862"
}
}
Loading