Skip to content

Latest commit

 

History

History
209 lines (143 loc) · 8.33 KB

notes001-019.md

File metadata and controls

209 lines (143 loc) · 8.33 KB

Topic 019: NodeJS fs and https-proxy-agent module

Topic 019: NodeJS fs and https-proxy-agent module

fs module

The fs (file system) module in Node.js provides various methods for interacting with the file system, including renaming and copying files. The renameSync and copyFileSync methods are used for these purposes, respectively.

renameSync

The renameSync method synchronously renames a file or directory. This is a blocking operation, meaning it will halt the execution of your program until the rename operation is complete.

Syntax:

fs.renameSync(oldPath, newPath);

Parameters:

  • oldPath: The current path of the file or directory.
  • newPath: The new path of the file or directory.

Example:

const fs = require("fs");

try {
  fs.renameSync("path/to/oldFilename.txt", "path/to/newFilename.txt");
  console.log("File renamed successfully");
} catch (err) {
  console.error("Error renaming file:", err);
}

copyFileSync

The copyFileSync method synchronously copies a file from the source path to the destination path. This is also a blocking operation.

Syntax:

fs.copyFileSync(src, dest[, mode])

Parameters:

  • src: The path of the source file.
  • dest: The path of the destination file.
  • mode (optional): An optional integer that specifies the behavior of the copy operation. Possible values include fs.constants.COPYFILE_EXCL to make the operation fail if the destination file exists, among others.

Example:

const fs = require("fs");

try {
  fs.copyFileSync("path/to/sourceFile.txt", "path/to/destinationFile.txt");
  console.log("File copied successfully");
} catch (err) {
  console.error("Error copying file:", err);
}

Notes

  • Blocking Operations: Both renameSync and copyFileSync are blocking operations, which means they will stop the execution of subsequent code until the operation completes. For non-blocking versions, you can use fs.rename and fs.copyFile respectively.
  • Error Handling: Always wrap these operations in a try-catch block to handle any potential errors that might occur during the file operations.

These methods are useful for simple file manipulations in scripts where synchronous behavior is acceptable or desirable. For more complex applications, especially those that require high performance or need to handle multiple concurrent operations, consider using the asynchronous versions of these methods.

https-proxy-agent module

The proxy=false setting in https-proxy-agent is used to disable proxying altogether, meaning that the https-proxy-agent will not attempt to use any proxy for the request. This can be useful in scenarios where proxy configurations might be set globally, but you need to ensure that a specific request bypasses the proxy and goes directly to the destination server.

Understanding https-proxy-agent

https-proxy-agent is a Node.js module that allows you to proxy HTTPS requests through an HTTP, HTTPS, or SOCKS proxy. When configuring the agent, you can provide various options, including the proxy URL and any necessary authentication.

Example of Using https-proxy-agent with proxy=false

Consider the following example where you configure https-proxy-agent to bypass the proxy:

const https = require("https");
const HttpsProxyAgent = require("https-proxy-agent");

// Assume you have a global proxy setting, e.g., process.env.HTTP_PROXY
const proxy = process.env.HTTP_PROXY || "http://example.com:8080";

// Create an instance of HttpsProxyAgent
const agent = new HttpsProxyAgent(proxy);

// Disable proxying for a specific request by setting proxy to false
const options = {
  hostname: "www.example.com",
  port: 443,
  path: "/",
  method: "GET",
  agent: false, // This disables the use of the proxy for this request
};

const req = https.request(options, (res) => {
  console.log(`STATUS: ${res.statusCode}`);
  console.log(`HEADERS: ${JSON.stringify(res.headers)}`);
  res.setEncoding("utf8");
  res.on("data", (chunk) => {
    console.log(`BODY: ${chunk}`);
  });
  res.on("end", () => {
    console.log("No more data in response.");
  });
});

req.on("error", (e) => {
  console.error(`Problem with request: ${e.message}`);
});

req.end();

Why Use proxy=false?

  1. Bypass Global Proxy Settings: If your environment has a global proxy configured (e.g., through environment variables like HTTP_PROXY or HTTPS_PROXY), setting proxy=false for specific requests ensures those requests bypass the proxy.

  2. Direct Access Needs: Sometimes, certain endpoints should not be accessed through a proxy, either for security reasons or to ensure direct connectivity. Using proxy=false helps achieve this.

  3. Custom Configuration: In more complex applications, you might need to dynamically decide whether to use a proxy or not based on the target URL or other request parameters. Setting proxy=false provides flexibility in such cases.

Note

  • The proxy=false setting is specific to the agent option in the https module. When agent is set to false, it means no agent (including any proxy agent) will be used, and the connection will be made directly.
  • This approach is different from configuring https-proxy-agent directly since the agent configuration would be bypassed entirely for that request.

By understanding how to control proxy settings at a per-request level, you can fine-tune the behavior of your Node.js application to suit different network environments and requirements.

YAML keyword replacement

Keyword replacement in a YAML file, particularly in the context of CI/CD (Continuous Integration/Continuous Deployment), typically involves dynamically substituting placeholder values with actual values during the execution of a CI/CD pipeline. This technique is often used to manage configuration settings, secrets, and environment-specific variables in a flexible and reusable manner.

Here’s a more detailed explanation:

  1. Placeholders in YAML: You define placeholders in your YAML configuration file. These placeholders are usually in a format that is easily recognizable and can be programmatically replaced. Common formats include {{PLACEHOLDER}}, ${PLACEHOLDER}, or %PLACEHOLDER%.

  2. Pipeline Variables: In your CI/CD pipeline (e.g., using tools like Jenkins, GitLab CI, GitHub Actions, Azure Pipelines, etc.), you define the actual values for these placeholders. These values can be defined as environment variables, pipeline parameters, or secrets.

  3. Replacement Process: During the pipeline execution, a step or script will replace the placeholders in the YAML file with the actual values. This can be done using built-in functions of the CI/CD tool, custom scripts, or specialized tools like envsubst, yq, or sed.

Example

Suppose you have a YAML configuration file config.yaml with the following content:

database:
  host: { { DB_HOST } }
  username: { { DB_USER } }
  password: { { DB_PASS } }

In your CI/CD pipeline, you would define the variables:

  • DB_HOST=database.example.com
  • DB_USER=admin
  • DB_PASS=secretpassword

During the pipeline execution, these placeholders will be replaced with the actual values:

database:
  host: database.example.com
  username: admin
  password: secretpassword

Using GitLab CI as an Example

.gitlab-ci.yml:

stages:
  - replace

variables:
  DB_HOST: "database.example.com"
  DB_USER: "admin"
  DB_PASS: "secretpassword"

replace_placeholders:
  stage: replace
  script:
    - sed -i 's/{{DB_HOST}}/'"$DB_HOST"'/g' config.yaml
    - sed -i 's/{{DB_USER}}/'"$DB_USER"'/g' config.yaml
    - sed -i 's/{{DB_PASS}}/'"$DB_PASS"'/g' config.yaml
  artifacts:
    paths:
      - config.yaml

In this example, sed is used to perform the replacement. The -i flag edits the file in place, and the s/.../.../g syntax performs the substitution.

Tools and Methods

  • envsubst: A simple utility to substitute environment variables in a shell script.
  • yq: A lightweight and portable command-line YAML processor.
  • sed: A stream editor for filtering and transforming text.
  • Custom Scripts: Written in shell, Python, or other scripting languages.

By using keyword replacement in YAML files, you ensure that your configuration is flexible and environment-agnostic, which is crucial for maintaining robust CI/CD pipelines.