Skip to content

WIP: export kaniko layers on run image #797

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jabrown85 wants to merge 7 commits into from
Closed

WIP: export kaniko layers on run image #797

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
d8e99a9
wip
jabrown85 Jan 20, 2022
87ff4de
fixup! wip
jabrown85 Jan 20, 2022
d97defe
store kaniko snapshots in different directories
jabrown85 Jan 20, 2022
1f2d774
more wip work
jabrown85 Jan 25, 2022
0f6fb26
fixup! more wip work
jabrown85 Jan 25, 2022
03a9503
even more work
jabrown85 Jan 25, 2022
9dd5d4b
turn off cache
jabrown85 Jan 25, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions exporter.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"encoding/json"
"fmt"
"io"
"io/ioutil"
"os"
"path/filepath"
"strings"
Expand Down Expand Up @@ -91,6 +92,12 @@ func (e *Exporter) Export(opts ExportOptions) (platform.ExportReport, error) {
return platform.ExportReport{}, errors.Wrap(err, "read build metadata")
}

// extender layers
// the new base is currently being referenced via run-image
// if err := e.addExtenderLayers(opts, &meta); err != nil {
// return platform.ExportReport{}, err
// }

// buildpack-provided layers
if err := e.addBuildpackLayers(opts, &meta); err != nil {
Copy link

@cmoulliard cmoulliard Feb 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this function only be used to extract the Run or Build/Run layers ? If the function will only be used to extract the Run layers, then it should be renamed ?

Copy link
Contributor Author

@jabrown85 jabrown85 Feb 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function already existed and only handles buildpack layers. I actually could remove the exporter changes and addExtenderLayers entirely. But I didn't do so yet depending on how we plan on moving forward. My plan was to assume a new run image reference came out of the run extension and was already pushed to the remote registry. We could alternatively have the run extensions save a run image as OCI Layout/Tarball and then have exporter build with those layers (or even push up the run image to the remote registry).

Copy link

@cmoulliard cmoulliard Feb 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add then a TODO to review later or better document what this step of the code is doing ?

return platform.ExportReport{}, err
Expand Down Expand Up @@ -284,6 +291,67 @@ func (e *Exporter) addAppLayers(opts ExportOptions, slices []layers.Slice, meta
return nil
}

func (e *Exporter) addExtenderLayers(opts ExportOptions, meta *platform.LayersMetadata) error {
e.Logger.Info("Adding extender layers")

path := filepath.Join("/layers-run/kaniko")
Copy link

@cmoulliard cmoulliard Feb 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you plan to extract the layers for every build(pack) build or do we plan to move the already extracted layers to a cache and reuse them for next build ?

dirs, err := ioutil.ReadDir(path)
if err != nil {
if os.IsNotExist(err) {
return nil
}
return err
}

for _, dir := range dirs {
if !dir.IsDir() {
continue
}

e.Logger.Infof("Adding extender layers for %s", dir.Name())
manifestPath := filepath.Join(path, dir.Name(), "manifest.json")
var manifest []struct {
Layers []string `json:"Layers"`
}

f, err := os.Open(manifestPath)
if err != nil {
if os.IsNotExist(err) {
return nil
}

return err
}

defer f.Close()

contents, _ := ioutil.ReadAll(f)
e.Logger.Infof("kaniko manifest: %s", string(contents))
f.Seek(0, 0)

err = json.NewDecoder(f).Decode(&manifest)
if err != nil {
return err
}

if len(manifest) == 0 || len(manifest[0].Layers) == 0 {
return nil
}

// Iterate layers except first one (the base layer)
for _, layerName := range manifest[0].Layers[1:] {
tarPath := filepath.Join(path, dir.Name(), layerName)
e.Logger.Infof("Adding extended layer '%s'", tarPath)
err := opts.WorkingImage.AddLayer(tarPath)
if err != nil {
return errors.Wrapf(err, "creating ext layer")
}
}
}

return nil
}

func (e *Exporter) setLabels(opts ExportOptions, meta platform.LayersMetadata, buildMD *platform.BuildMetadata) error {
data, err := json.Marshal(meta)
if err != nil {
Expand Down
159 changes: 90 additions & 69 deletions extender/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,14 @@ package main
import (
"encoding/json"
"fmt"
"github.com/BurntSushi/toml"
"github.com/GoogleContainerTools/kaniko/pkg/config"
cfg "github.com/redhat-buildpacks/poc/kaniko/buildpackconfig"
"github.com/redhat-buildpacks/poc/kaniko/util"
"os"
"os/exec"
"path"
"path/filepath"
"syscall"

"github.com/BurntSushi/toml"
"github.com/GoogleContainerTools/kaniko/pkg/config"
cfg "github.com/redhat-buildpacks/poc/kaniko/buildpackconfig"
)

type BuildMetadata struct {
Expand Down Expand Up @@ -61,6 +60,7 @@ func doKaniko(kind, baseimage string) {
fmt.Println("Initialize the BuildPackConfig and set the defaults values ...")
b := cfg.NewBuildPackConfig()
b.InitDefaults()

b.ExtractLayers = false

fmt.Printf("Kaniko dir: %s\n", b.KanikoDir)
Expand All @@ -76,30 +76,91 @@ func doKaniko(kind, baseimage string) {
panic(err)
}

toMultiArg := func(args []DockerfileArg) []string {
var result []string
for _, arg := range args {
result = append(result, fmt.Sprintf("%s=%s", arg.Key, arg.Value))
// start with the base image given to us by the builder
current_base_image := baseimage

for _, d := range meta.Dockerfiles {
// skip the extensions not of this type
if d.Type != kind {
continue
}

result = append(result, fmt.Sprintf(`base_image=%s`, baseimage))
return result
}
toMultiArg := func(args []DockerfileArg) []string {
var result []string
for _, arg := range args {
result = append(result, fmt.Sprintf("%s=%s", arg.Key, arg.Value))
}

result = append(result, fmt.Sprintf(`base_image=%s`, current_base_image))
return result
}

if kind == "build" {
// TODO: execute all but the FROM clauses in the Dockerfile?
// Define opts
opts := config.KanikoOptions{
BuildArgs: toMultiArg(d.Args),
DockerfilePath: d.Path,
// Cache: true,
// CacheOptions: config.CacheOptions{
// CacheDir: b.CacheDir,
// },
IgnoreVarRun: true,
NoPush: true,
SrcContext: b.WorkspaceDir,
SnapshotMode: "full",
// OCILayoutPath: "/layers/kaniko",
// ImageNameDigestFile: fmt.Sprintf("/layers/kaniko/idk%s", b.Destination),
RegistryOptions: config.RegistryOptions{
SkipTLSVerify: true,
},
// IgnorePaths: b.IgnorePaths,
// TODO: we can not output a tar for intermediate images, we only need the last one
// TarPath: "/layers/kaniko/new_base.tar",
// Destinations: []string{b.Destination},
// ForceBuildMetadata: true,
Cleanup: false,
}

// Build the Dockerfile
fmt.Printf("Building the %s\n", opts.DockerfilePath)
err := b.BuildDockerFile(opts)
if err != nil {
panic(err)
}
continue
}

// run extensions

// we need a registry right now, because kaniko is pulling the image to build on top of for subsequent Dockerfile exts
registryHost := os.Getenv("REGISTRY_HOST")
b.Destination = fmt.Sprintf("%s/extended/runimage", registryHost)
fmt.Printf("Destination Image: %s\n", b.Destination)

for _, d := range meta.Dockerfiles {
// Define opts
opts := config.KanikoOptions{
BuildArgs: toMultiArg(d.Args),
DockerfilePath: d.Path,
CacheOptions: config.CacheOptions{CacheDir: b.CacheDir},
IgnoreVarRun: true,
NoPush: true,
SrcContext: b.WorkspaceDir,
SnapshotMode: "full",
IgnorePaths: b.IgnorePaths,
TarPath: b.LayerTarFileName,
Destinations: []string{b.Destination},
ForceBuildMetadata: true,
BuildArgs: toMultiArg(d.Args),
DockerfilePath: d.Path,
// Cache: true,
// CacheOptions: config.CacheOptions{
// CacheDir: b.CacheDir,
// },
IgnoreVarRun: true,
// NoPush: true,
SrcContext: b.WorkspaceDir,
SnapshotMode: "full",
// OCILayoutPath: "/layers/kaniko",
// ImageNameDigestFile: fmt.Sprintf("/layers/kaniko/idk%s", b.Destination),
// IgnorePaths: b.IgnorePaths,
// TODO: we can not output a tar for intermediate images, we only need the last one
// TarPath: "/layers/kaniko/new_base.tar",
Destinations: []string{b.Destination},
// ForceBuildMetadata: true,
Cleanup: true,
RegistryOptions: config.RegistryOptions{
SkipTLSVerify: true,
},
}

// Build the Dockerfile
Expand All @@ -108,53 +169,13 @@ func doKaniko(kind, baseimage string) {
if err != nil {
panic(err)
}
}

// Log the content of the Kaniko dir
fmt.Printf("Reading dir content of: %s\n", b.KanikoDir)
util.ReadFilesFromPath(b.KanikoDir)

// TODO: caching doesn't seem to be working at the moment. Need to investigate...
// Copy the tgz layer file to the Cache dir
srcPath := path.Join("/", b.LayerTarFileName)
dstPath := path.Join(b.CacheDir, b.LayerTarFileName)

// Ensure cache directory exists
fmt.Printf("Creating %s dir ...\n", b.CacheDir)
err = os.MkdirAll(b.CacheDir, os.ModePerm)
if err != nil {
panic(err)
}

fmt.Printf("Copy the %s file to the %s dir ...\n", srcPath, dstPath)
err = util.File(srcPath, dstPath)
if err != nil {
panic(err)
}

fmt.Printf("Extract the content of the tarball file %s under the cache %s\n", b.Opts.TarPath, b.Opts.CacheDir)
err = untar(dstPath, b.CacheDir)
if err != nil {
panic(err)
}

layerFiles, err := layerTarfiles()
if err != nil {
panic(err)
// TODO: next base image is the one we just built, we should use digest instead
current_base_image = b.Destination
}

// run buildpacks now
if kind == "build" {
// We're in "build" mode, untar layers to root filesystem: /
for _, layerFile := range layerFiles {
workingDirectory := "/"
tarPath := "/layers/kaniko/" + layerFile
Copy link

@cmoulliard cmoulliard Feb 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should maybe avoid to hard code kaniko but pass it as parameter as it could be possible that another solution could be supported in the future (buildah, ...)


err = untar(tarPath, workingDirectory)
if err != nil {
panic(err)
}
}

// Run the build for buildpacks with lowered privileges.
// We must assume that this extender is run as root.
cmd := exec.Command("/cnb/lifecycle/builder", "-app", "/workspace", "-log-level", "debug")
Copy link

@cmoulliard cmoulliard Feb 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you plan to log an error message for the end user to be informed that a root privileged are needed ?

Remark: Why do you say that Run the build for buildpacks with lowered privileges. as comment ?

Copy link
Contributor Author

@jabrown85 jabrown85 Feb 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the first question, I do assume we will need to introduce "must run as root" error message like we do elsewhere (creator does this). I didn't add the comment you are asking about, but the idea is that we want to run normal buildpacks after build extensions but in the same process. The /cnb/lifecycle/builder should refuse to run as root, so the ext binary will need to ensure they aren't running as root.

Expand All @@ -177,8 +198,8 @@ func untar(tarPath, workingDirectory string) error {
return command.Run()
}

func layerTarfiles() ([]string, error) {
manifestPath := "/layers/kaniko/manifest.json"
func layerTarfiles(manifestPath string) ([]string, error) {
// manifestPath := "/layers/kaniko/manifest.json"
var manifest []struct {
Layers []string `json:"Layers"`
}
Expand Down
Loading