diff --git a/docs/README.md b/docs/README.md index c9ba0e8b..4524d98e 100644 --- a/docs/README.md +++ b/docs/README.md @@ -11,6 +11,14 @@ Welcome to the Hemlock programming language documentation! - [Quick Start](getting-started/quick-start.md) - Your first Hemlock program - [Tutorial](getting-started/tutorial.md) - Step-by-step guide to Hemlock basics +### Learning Systems Programming +- [Education Overview](education/README.md) - Learning systems programming with Hemlock +- [Why Manual Memory Matters](education/why-memory-matters.md) - Conceptual foundation +- [From Python/JS to Systems](education/from-high-level.md) - Bridge guide for high-level programmers +- [Common Mistakes](education/common-mistakes.md) - Debugging memory and concurrency bugs +- [Learning Curriculum](education/curriculum.md) - Structured learning path +- [Exercises](education/exercises/) - Hands-on practice exercises + ### Language Guide - [Syntax Overview](language-guide/syntax.md) - Basic syntax and structure - [Type System](language-guide/types.md) - Primitive types, type inference, and conversions @@ -108,6 +116,7 @@ We give you the tools to be safe (`buffer`, type annotations, bounds checking) b - **Package Manager**: [hpm](https://github.com/hemlang/hpm) - Hemlock Package Manager - **Issues**: Report bugs and request features - **Examples**: See the `examples/` directory +- **Educational Examples**: See `examples/educational/` for learning-focused programs - **Tests**: See the `tests/` directory for usage examples ## License diff --git a/docs/education/README.md b/docs/education/README.md new file mode 100644 index 00000000..c604fe7d --- /dev/null +++ b/docs/education/README.md @@ -0,0 +1,297 @@ +# Learning Systems Programming with Hemlock + +> "Hemlock exists to teach you what your computer is actually doing." + +This educational guide positions Hemlock as a **teaching language for systems programming concepts**. Whether you're coming from Python, JavaScript, or another high-level language, Hemlock provides a gentle but honest introduction to how computers actually manage memory and execute code. + +--- + +## Why Hemlock for Learning? + +### The Problem with Learning C + +C is the traditional path to systems programming, but it presents several pedagogical challenges: + +1. **Compilation complexity** - Header files, makefiles, linker errors distract from concepts +2. **Cryptic errors** - Segfaults give no context; debugging requires external tools +3. **No guardrails** - Undefined behavior can silently corrupt memory +4. **Boilerplate overhead** - `#include`, `main()` signatures, type declarations everywhere + +### The Problem with Staying High-Level + +Python and JavaScript hide fundamental concepts: + +1. **Memory is invisible** - Garbage collection prevents understanding allocation +2. **Types are hidden** - Dynamic typing masks how data is represented +3. **Pointers don't exist** - References are abstracted away +4. **Performance is magic** - No model for why code is fast or slow + +### Hemlock: The Middle Path + +Hemlock gives you **manual control with modern ergonomics**: + +```hemlock +// Simple syntax, explicit memory +let p = alloc(64); // You allocate +memset(p, 0, 64); // You initialize +// ... use memory ... +free(p); // You clean up +``` + +**What Hemlock offers:** +- **Scripting syntax** - No headers, no makefiles, just `.hml` files +- **Helpful errors** - Runtime errors include context and line numbers +- **Optional safety** - Use `buffer` for bounds checking, `ptr` for raw access +- **Dual backends** - Interpret for rapid iteration, compile for understanding generated code +- **Real parallelism** - pthreads under the hood, not green threads + +--- + +## The Hemlock Learning Philosophy + +### 1. Make the Invisible Visible + +High-level languages hide memory. Hemlock makes it explicit: + +```hemlock +// In Python: x = [1, 2, 3] -- Where does this live? How big is it? + +// In Hemlock: You decide +let arr = [1, 2, 3]; // Dynamic array (managed) +let p = alloc(3 * 4); // Raw memory: 3 integers × 4 bytes +ptr_write_i32(p, 1); // You write each value +ptr_write_i32(p + 4, 2); // Pointer arithmetic is explicit +ptr_write_i32(p + 8, 3); +free(p); // You clean up +``` + +### 2. Unsafe is Educational + +Hemlock lets you make mistakes safely: + +```hemlock +// This WILL crash - and that's the lesson +let p = alloc(8); +free(p); +free(p); // Double-free: crashes with error message +``` + +In C, this might silently corrupt memory. In Hemlock, you get an error you can learn from. + +### 3. Graduate from Training Wheels + +Hemlock provides a safety spectrum: + +```hemlock +// Level 1: Managed arrays (safest) +let arr = [1, 2, 3]; +arr[10]; // Runtime error: Index out of bounds + +// Level 2: Bounds-checked buffers +let b = buffer(64); +b[100]; // Runtime error: Buffer bounds exceeded + +// Level 3: Raw pointers (full control) +let p = alloc(64); +// p[100] - No checking. Your responsibility. Real systems programming. +``` + +### 4. Types Tell Stories + +Every value in Hemlock carries its type at runtime: + +```hemlock +let x = 42; +print(typeof(x)); // "i32" - 32-bit signed integer + +let y = 5000000000; +print(typeof(y)); // "i64" - Too big for i32, auto-promoted + +let z = 3.14; +print(typeof(z)); // "f64" - Floating point +``` + +This teaches you to think about data representation. + +--- + +## Learning Tracks + +### Track 1: Memory Fundamentals (Start Here) +*For programmers from GC languages* + +1. [Why Manual Memory Matters](./why-memory-matters.md) +2. [Your First Allocation](./exercises/01-first-allocation.md) +3. [The Stack vs The Heap](./exercises/02-stack-vs-heap.md) +4. [Pointers Are Just Numbers](./exercises/03-pointers-explained.md) +5. [Common Memory Mistakes](./common-mistakes.md) + +### Track 2: Types and Representation +*Understanding how data lives in memory* + +1. [Numeric Types and Their Ranges](./exercises/04-numeric-types.md) +2. [Type Promotion and Coercion](./exercises/05-type-promotion.md) +3. [Strings Are Byte Arrays](./exercises/06-strings-as-bytes.md) +4. [Structs and Memory Layout](./exercises/07-memory-layout.md) + +### Track 3: Concurrent Thinking +*Real parallelism, real problems* + +1. [Tasks and Threads](./exercises/08-tasks-intro.md) +2. [Channels for Communication](./exercises/09-channels.md) +3. [Race Conditions (On Purpose)](./exercises/10-race-conditions.md) +4. [Atomics and Lock-Free Code](./exercises/11-atomics.md) + +### Track 4: Systems Integration +*Connecting to the real world* + +1. [Calling C Functions (FFI)](./exercises/12-ffi-basics.md) +2. [Working with Files](./exercises/13-file-io.md) +3. [Handling Signals](./exercises/14-signals.md) +4. [Building Real Tools](./exercises/15-real-tools.md) + +--- + +## Quick Comparison: Python vs Hemlock + +| Concept | Python | Hemlock | +|---------|--------|---------| +| Create a list | `x = [1, 2, 3]` | `let x = [1, 2, 3];` | +| Allocate memory | *(hidden)* | `let p = alloc(64);` | +| Free memory | *(GC does it)* | `free(p);` | +| Get a pointer | *(not possible)* | `let p = &x;` | +| Type of value | `type(x)` | `typeof(x)` | +| String length | `len(s)` | `len(s)` or `s.length` | +| Check bounds | *(automatic)* | Use `buffer` or check manually | +| Run in parallel | `threading`/`multiprocessing` | `spawn(fn)` (real threads) | +| Call C code | `ctypes`/`cffi` | `extern fn` or `ffi_bind` | + +--- + +## Pedagogical Features + +### 1. Introspection Everywhere + +```hemlock +// Always know what you're working with +let x = 42; +print(typeof(x)); // "i32" +print(sizeof(i32)); // 4 (bytes) + +let arr = [1, 2, 3]; +print(len(arr)); // 3 +print(typeof(arr)); // "array" +``` + +### 2. Explicit Type Annotations (Optional) + +```hemlock +// Types are checked but not required +let x = 42; // Type inferred as i32 +let y: i64 = 42; // Explicitly i64 +let z: i32 = "hello"; // Runtime error: Type mismatch +``` + +### 3. Memory Tracking + +```hemlock +import { memory_stats } from "@stdlib/env"; + +let p1 = alloc(100); +let p2 = alloc(200); + +let stats = memory_stats(); +print("Allocated: " + stats.allocated + " bytes"); +print("Allocations: " + stats.allocations); + +free(p1); +free(p2); +``` + +### 4. Helpful Runtime Errors + +```hemlock +// Instead of "Segmentation fault (core dumped)" +// You get: + +// Error at line 15: Double-free detected +// Error at line 23: Index 10 out of bounds for array of length 3 +// Error at line 31: Null pointer dereference +// Error at line 42: Type mismatch: expected i32, got string +``` + +--- + +## Setting Up Your Learning Environment + +### Installation + +```bash +git clone https://github.com/hemlang/hemlock.git +cd hemlock +make +``` + +### Running Programs + +```bash +# Interpret (fast iteration) +./hemlock program.hml + +# Compile (see generated C) +./hemlockc program.hml -o program --keep-c +cat program.c # Examine generated code +./program +``` + +### Recommended Workflow + +1. **Write** your `.hml` file +2. **Interpret** first to verify correctness +3. **Compile** to see the C translation +4. **Read** the generated C to understand what happens + +--- + +## The Journey + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Your Learning Journey │ +├─────────────────────────────────────────────────────────────────┤ +│ │ +│ Python/JS ──► Hemlock (managed) ──► Hemlock (raw) ──► C │ +│ │ +│ "I don't "I see where "I control "I understand │ +│ think about memory lives" every byte" everything" │ +│ memory" │ +│ │ +└─────────────────────────────────────────────────────────────────┘ +``` + +Hemlock is a **bridge language**. It's not meant to replace C for production systems work. It's meant to teach you the concepts that make C comprehensible. + +--- + +## Next Steps + +- **New to systems programming?** Start with [Why Manual Memory Matters](./why-memory-matters.md) +- **Coming from Python/JS?** Read [From High-Level to Systems](./from-high-level.md) +- **Want exercises?** Jump to [Memory Exercises](./exercises/) +- **Just want to code?** See the [examples/](../../examples/) directory + +--- + +## Educational Resources + +| Resource | Description | +|----------|-------------| +| [exercises/](./exercises/) | Hands-on exercises with solutions | +| [common-mistakes.md](./common-mistakes.md) | Mistakes and how to debug them | +| [why-memory-matters.md](./why-memory-matters.md) | Conceptual foundation | +| [from-high-level.md](./from-high-level.md) | Bridge guide for Python/JS devs | +| [curriculum.md](./curriculum.md) | Structured learning path | + +--- + +*"The goal isn't to make systems programming easy. The goal is to make it understandable."* diff --git a/docs/education/common-mistakes.md b/docs/education/common-mistakes.md new file mode 100644 index 00000000..692983fc --- /dev/null +++ b/docs/education/common-mistakes.md @@ -0,0 +1,730 @@ +# Common Mistakes and How to Debug Them + +> "Every crash is a lesson. Every bug is a teacher." + +This guide covers the most common mistakes when learning systems programming with Hemlock, why they happen, how to recognize them, and how to fix them. + +--- + +## Table of Contents + +1. [Memory Mistakes](#memory-mistakes) + - [Double Free](#1-double-free) + - [Use After Free](#2-use-after-free) + - [Memory Leak](#3-memory-leak) + - [Buffer Overflow](#4-buffer-overflow) + - [Dangling Pointer](#5-dangling-pointer) + - [Null Pointer Dereference](#6-null-pointer-dereference) + - [Uninitialized Memory](#7-uninitialized-memory) +2. [Type Mistakes](#type-mistakes) + - [Integer Overflow](#8-integer-overflow) + - [Type Mismatch](#9-type-mismatch) + - [Precision Loss](#10-precision-loss) +3. [Concurrency Mistakes](#concurrency-mistakes) + - [Race Condition](#11-race-condition) + - [Use After Spawn](#12-use-after-spawn) + - [Deadlock](#13-deadlock) +4. [Logic Mistakes](#logic-mistakes) + - [Off-by-One Error](#14-off-by-one-error) + - [Integer Division](#15-integer-division) + - [Forgetting Defer](#16-forgetting-defer) + +--- + +## Memory Mistakes + +### 1. Double Free + +**What it is:** Freeing the same memory twice. + +**Example:** +```hemlock +let p = alloc(64); +// ... use p ... +free(p); + +// Later, forgetting it was freed... +free(p); // CRASH: Double free! +``` + +**Why it's bad:** The memory allocator's internal bookkeeping gets corrupted. This can cause crashes, data corruption, or security vulnerabilities. + +**Symptoms:** +- Runtime error: "Double-free detected" +- In C: Crash with no message, or exploitable vulnerability + +**How to fix:** +```hemlock +// Option 1: Set to null after free +free(p); +p = ptr_null(); + +// Now double-free is detectable +if (p != ptr_null()) { + free(p); +} + +// Option 2: Track ownership clearly +fn destroy_thing(p: ptr) { + // Document: This function takes ownership and frees + free(p); +} +``` + +**Prevention:** +- Set pointers to `ptr_null()` immediately after freeing +- Document ownership: who allocates, who frees +- Use `defer` for cleanup + +--- + +### 2. Use After Free + +**What it is:** Accessing memory that has been freed. + +**Example:** +```hemlock +let p = alloc(64); +ptr_write_i32(p, 42); +free(p); + +// Memory is freed but pointer still exists +let x = ptr_read_i32(p); // UNDEFINED BEHAVIOR! +print(x); // Might print 42, garbage, or crash +``` + +**Why it's bad:** The freed memory may be reallocated for something else. Reading gives garbage; writing corrupts other data. + +**Symptoms:** +- Random crashes +- Corrupted data +- "Impossible" values appearing +- Works sometimes, fails other times + +**How to fix:** +```hemlock +// Always nullify after free +free(p); +p = ptr_null(); + +// Check before use +if (p != ptr_null()) { + let x = ptr_read_i32(p); +} +``` + +--- + +### 3. Memory Leak + +**What it is:** Allocating memory but never freeing it. + +**Example:** +```hemlock +fn process_data() { + let p = alloc(1000); + // ... do work ... + + if (error_occurred()) { + return; // LEAK! Forgot to free p + } + + free(p); +} +``` + +**Why it's bad:** Memory accumulates over time. Long-running programs eventually exhaust system memory. + +**Symptoms:** +- Memory usage grows continuously +- Program slows down over time +- Eventually: "Out of memory" error + +**How to fix:** +```hemlock +// Use defer for automatic cleanup +fn process_data() { + let p = alloc(1000); + defer free(p); // Will run no matter how function exits + + if (error_occurred()) { + return; // defer handles cleanup + } + + // ... more work ... +} // defer runs here too +``` + +**Prevention:** +- Use `defer` for all allocations +- Match every `alloc` with a `free` +- Use tracking tools to detect leaks + +--- + +### 4. Buffer Overflow + +**What it is:** Writing past the end of allocated memory. + +**Example:** +```hemlock +let p = alloc(10); // Only 10 bytes + +// Writing beyond bounds +for (let i = 0; i < 20; i++) { + ptr_write_u8(p + i, i); // Overflow when i >= 10! +} + +free(p); +``` + +**Why it's bad:** Overwrites adjacent memory, corrupting data or control structures. The #1 source of security vulnerabilities. + +**Symptoms:** +- Crashes +- Corrupted data in unrelated variables +- Security exploits (in real-world code) + +**How to fix:** +```hemlock +// Option 1: Use buffer for bounds checking +let b = buffer(10); +b[15] = 42; // Runtime error: "Index out of bounds" + +// Option 2: Check bounds manually +let size = 10; +let p = alloc(size); +for (let i = 0; i < 20; i++) { + if (i < size) { + ptr_write_u8(p + i, i); + } else { + print("Warning: skipping out-of-bounds write"); + } +} +free(p); +``` + +--- + +### 5. Dangling Pointer + +**What it is:** A pointer to memory that no longer belongs to you. + +**Example:** +```hemlock +fn get_local_address(): ptr { + let x = 42; // x is on the stack + return &x; // Return address of stack variable +} // x is destroyed here! + +let p = get_local_address(); +let value = ptr_read_i32(p); // DANGLING: x no longer exists +``` + +**Why it's bad:** Stack memory is reused. Your pointer now points to some other function's variables or garbage. + +**Symptoms:** +- Random values +- Values that change unexpectedly +- Crashes when dereferencing + +**How to fix:** +```hemlock +// Option 1: Return the value, not a pointer +fn get_value(): i32 { + let x = 42; + return x; // Value is copied out +} + +// Option 2: Allocate on heap (caller must free) +fn get_heap_value(): ptr { + let p = alloc(sizeof(i32)); + ptr_write_i32(p, 42); + return p; +} +``` + +--- + +### 6. Null Pointer Dereference + +**What it is:** Trying to read or write through a null pointer. + +**Example:** +```hemlock +fn find_item(key: string): ptr { + // Returns null if not found + return ptr_null(); +} + +let p = find_item("missing"); +let value = ptr_read_i32(p); // CRASH: Null dereference! +``` + +**Why it's bad:** Address 0 is never valid memory. Dereferencing it crashes the program. + +**Symptoms:** +- Immediate crash +- "Segmentation fault" in C +- Runtime error in Hemlock + +**How to fix:** +```hemlock +let p = find_item("missing"); +if (p == ptr_null()) { + print("Item not found"); +} else { + let value = ptr_read_i32(p); + print("Found: " + value); +} +``` + +--- + +### 7. Uninitialized Memory + +**What it is:** Reading memory before writing to it. + +**Example:** +```hemlock +let p = alloc(100); +// Forgot to initialize! +let x = ptr_read_i32(p); // Reading garbage +print(x); // Unpredictable value +``` + +**Why it's bad:** Uninitialized memory contains whatever was there before - could be zeros, old data, or garbage. + +**Symptoms:** +- Different results each run +- "Impossible" values +- Works on your machine, fails elsewhere + +**How to fix:** +```hemlock +// Option 1: Zero-initialize with memset +let p = alloc(100); +memset(p, 0, 100); + +// Option 2: Initialize immediately +let p = alloc(sizeof(i32)); +ptr_write_i32(p, 0); // Explicit initialization + +// Option 3: Use talloc (zeroed allocation) +let p = talloc(i32, 25); // 25 zeroed i32s +``` + +--- + +## Type Mistakes + +### 8. Integer Overflow + +**What it is:** Arithmetic that exceeds a type's range. + +**Example:** +```hemlock +let x: i8 = 127; // i8 max is 127 +x = x + 1; // Overflow! Wraps to -128 +print(x); // -128 +``` + +**Why it's bad:** Values wrap around unexpectedly. Can cause infinite loops, security bugs, or corrupted calculations. + +**Symptoms:** +- Negative numbers appearing from positive arithmetic +- Infinite loops +- Wrong calculations + +**How to fix:** +```hemlock +// Use a larger type +let x: i32 = 127; +x = x + 1; +print(x); // 128 + +// Or check before operating +let x: i8 = 127; +if (x < 127) { + x = x + 1; +} else { + print("Would overflow!"); +} +``` + +--- + +### 9. Type Mismatch + +**What it is:** Using a value as the wrong type. + +**Example:** +```hemlock +let p = alloc(4); +ptr_write_i32(p, 42); + +// Read as wrong type +let f = ptr_read_f32(p); // Interprets integer bits as float! +print(f); // Some weird float value, not 42.0 +``` + +**Why it's bad:** Memory is just bytes. Interpretation depends on how you read it. + +**Symptoms:** +- Completely wrong values +- NaN or infinity in floats +- Garbage when reading strings + +**How to fix:** +```hemlock +// Be consistent in read/write types +let p = alloc(4); +ptr_write_f32(p, 42.0); +let f = ptr_read_f32(p); // Correct +print(f); // 42.0 +``` + +--- + +### 10. Precision Loss + +**What it is:** Converting between types that can't represent the same values. + +**Example:** +```hemlock +let big: i64 = 9007199254740993; // Larger than f64 can represent exactly +let f: f64 = big; // Precision lost! +print(f); // 9007199254740992 (off by 1) +``` + +**Why it's bad:** Silent data corruption. Looks correct but isn't. + +**Symptoms:** +- Off-by-one errors in large numbers +- Financial calculations going wrong +- IDs that don't match + +**How to fix:** +```hemlock +// Keep types consistent +let big: i64 = 9007199254740993; +// Don't convert to float if you need exact precision + +// Use string for display +print("Big number: " + big); +``` + +--- + +## Concurrency Mistakes + +### 11. Race Condition + +**What it is:** Two tasks accessing shared data without synchronization. + +**Example:** +```hemlock +let counter = 0; + +async fn increment() { + for (let i = 0; i < 1000; i++) { + counter = counter + 1; // RACE: read-modify-write is not atomic + } +} + +let t1 = spawn(increment); +let t2 = spawn(increment); +join(t1); +join(t2); + +print(counter); // Often NOT 2000! +``` + +**Why it's bad:** Both tasks read the same value, add 1, and write back. Some increments are lost. + +**Symptoms:** +- Non-deterministic results +- Lost updates +- Intermittent failures + +**How to fix:** +```hemlock +// Option 1: Use atomic operations +let p = alloc(sizeof(i32)); +atomic_store_i32(p, 0); + +async fn increment() { + for (let i = 0; i < 1000; i++) { + atomic_add_i32(p, 1); // Atomic: no race + } +} + +// Option 2: Use channels for communication +let ch = channel(1); +// Send updates through channel instead of shared state +``` + +--- + +### 12. Use After Spawn + +**What it is:** Freeing memory while a spawned task is still using it. + +**Example:** +```hemlock +async fn worker(data: ptr) { + sleep(100); // Simulating work + print(ptr_read_i32(data)); // CRASH: data is freed! +} + +let p = alloc(4); +ptr_write_i32(p, 42); +spawn(worker, p); +free(p); // Freed while worker is still running! +``` + +**Why it's bad:** The spawned task has a reference to memory you freed. + +**Symptoms:** +- Crashes in async code +- Corrupted data +- Intermittent failures + +**How to fix:** +```hemlock +// Wait for task before freeing +let p = alloc(4); +ptr_write_i32(p, 42); +let task = spawn(worker, p); +join(task); // Wait for completion +free(p); // Safe now +``` + +--- + +### 13. Deadlock + +**What it is:** Tasks waiting on each other forever. + +**Example:** +```hemlock +let ch1 = channel(0); // Unbuffered +let ch2 = channel(0); + +async fn task1() { + ch1.recv(); // Wait for ch1 + ch2.send("done"); // Then send on ch2 +} + +async fn task2() { + ch2.recv(); // Wait for ch2 (but task1 is waiting for ch1!) + ch1.send("done"); +} + +spawn(task1); +spawn(task2); +// DEADLOCK: Both tasks waiting forever +``` + +**Why it's bad:** Program hangs forever with no error message. + +**Symptoms:** +- Program hangs +- No output +- CPU at 0% + +**How to fix:** +```hemlock +// Establish a clear order +// OR use buffered channels +let ch1 = channel(1); // Buffered: send doesn't block if empty +let ch2 = channel(1); + +// OR use timeout mechanisms +// OR restructure to avoid circular dependencies +``` + +--- + +## Logic Mistakes + +### 14. Off-by-One Error + +**What it is:** Loop bounds or index calculations off by one. + +**Example:** +```hemlock +let arr = [1, 2, 3, 4, 5]; + +// WRONG: Accessing arr[5] which doesn't exist +for (let i = 0; i <= len(arr); i++) { + print(arr[i]); // Crashes on last iteration +} +``` + +**Symptoms:** +- Index out of bounds errors +- Missing first or last element +- Extra iteration + +**How to fix:** +```hemlock +// Use < not <= +for (let i = 0; i < len(arr); i++) { + print(arr[i]); +} + +// Or use for-in +for (x in arr) { + print(x); +} +``` + +--- + +### 15. Integer Division + +**What it is:** Using `/` expecting integer result. + +**Example:** +```hemlock +let a = 7; +let b = 2; +let result = a / b; +print(result); // 3.5 (float!) not 3 +``` + +**In Hemlock:** The `/` operator always returns a float! + +**How to fix:** +```hemlock +// Use divi for integer division +let result = divi(a, b); +print(result); // 3 + +// Or cast the float result +let result = i32(a / b); +print(result); // 3 +``` + +--- + +### 16. Forgetting Defer + +**What it is:** Returning early without cleaning up. + +**Example:** +```hemlock +fn process_file(path: string) { + let f = open(path, "r"); + + if (!validate(path)) { + return; // LEAK: File not closed! + } + + let data = f.read(); + f.close(); + return data; +} +``` + +**How to fix:** +```hemlock +fn process_file(path: string) { + let f = open(path, "r"); + defer f.close(); // Will ALWAYS run + + if (!validate(path)) { + return; // defer handles close + } + + return f.read(); +} // defer runs here too +``` + +--- + +## Debugging Strategies + +### 1. Add Print Statements + +```hemlock +print("DEBUG: Entering function"); +print("DEBUG: p = " + p); +print("DEBUG: value = " + ptr_read_i32(p)); +``` + +### 2. Check Assumptions + +```hemlock +fn process(p: ptr, size: i32) { + // Verify inputs + if (p == ptr_null()) { + panic("process: p is null"); + } + if (size <= 0) { + panic("process: size must be positive, got " + size); + } + // ... proceed ... +} +``` + +### 3. Use Assertions + +```hemlock +import { assert, assert_eq } from "@stdlib/assert"; + +assert(p != ptr_null(), "Pointer should not be null"); +assert_eq(len(arr), 5, "Array should have 5 elements"); +``` + +### 4. Isolate the Problem + +- Comment out code until the bug disappears +- Add code back until it reappears +- That's where the bug lives + +### 5. Compile and Inspect + +```bash +./hemlockc program.hml -o program --keep-c +cat program.c # See what the compiler generated +``` + +--- + +## Quick Reference: Error Messages + +| Error | Cause | Fix | +|-------|-------|-----| +| "Double-free detected" | Freeing same pointer twice | Null pointer after free | +| "Index out of bounds" | Array/buffer access beyond size | Check index < length | +| "Null pointer dereference" | Reading/writing through null | Check `p != ptr_null()` | +| "Type mismatch" | Wrong type in annotation | Fix type or conversion | +| "Division by zero" | Dividing by 0 | Check divisor before dividing | + +--- + +## Summary + +**Memory rules:** +1. Every `alloc` needs exactly one `free` +2. Never use memory after freeing it +3. Never return pointers to stack variables +4. Always check for null before dereferencing +5. Initialize memory before reading it + +**Concurrency rules:** +1. Protect shared data with atomics or channels +2. Wait for tasks before freeing their data +3. Avoid circular wait patterns + +**General rules:** +1. Use `defer` for cleanup +2. Check bounds before indexing +3. Use `divi()` for integer division +4. Validate assumptions early + +--- + +*"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it."* — Brian Kernighan diff --git a/docs/education/curriculum.md b/docs/education/curriculum.md new file mode 100644 index 00000000..1ececf83 --- /dev/null +++ b/docs/education/curriculum.md @@ -0,0 +1,548 @@ +# Hemlock Learning Curriculum + +A structured learning path for mastering systems programming concepts through Hemlock. + +--- + +## Who Is This For? + +This curriculum is designed for programmers who: +- Know at least one high-level language (Python, JavaScript, Ruby, etc.) +- Want to understand how computers actually manage memory +- Are curious about systems programming but intimidated by C +- Want to build a foundation for learning Rust, C, or low-level programming + +--- + +## How to Use This Curriculum + +1. **Follow the modules in order** - Each builds on the previous +2. **Do the exercises** - Reading isn't enough; you need hands-on practice +3. **Run the code** - Type it yourself, don't just copy-paste +4. **Make mistakes** - Crashes are learning opportunities +5. **Use both backends** - Interpret for quick iteration, compile to see generated C + +--- + +## Module 0: Getting Started + +**Duration:** 1-2 hours + +### Learning Objectives +- Install Hemlock +- Write and run your first program +- Understand the interpreter vs compiler + +### Content +1. **Installation** + ```bash + git clone https://github.com/hemlang/hemlock.git + cd hemlock + make + ``` + +2. **Hello World** + ```hemlock + // hello.hml + print("Hello, Systems Programming!"); + ``` + +3. **Running Programs** + ```bash + ./hemlock hello.hml # Interpret + ./hemlockc hello.hml -o hello # Compile + ./hello # Run compiled + ``` + +### Exercises +- [ ] Install Hemlock successfully +- [ ] Write a program that prints your name +- [ ] Compile the program and examine the binary + +### Resources +- [Installation Guide](../getting-started/installation.md) +- [Quick Start](../getting-started/quick-start.md) + +--- + +## Module 1: Hemlock Syntax Basics + +**Duration:** 2-3 hours + +### Learning Objectives +- Variables, types, and operators +- Control flow (if, while, for) +- Functions and closures +- Arrays and objects + +### Content + +**Variables and Types:** +```hemlock +let x = 42; // i32 (inferred) +let y: i64 = 100; // i64 (explicit) +let name = "Alice"; // string +let active = true; // bool +let pi = 3.14159; // f64 +``` + +**Control Flow:** +```hemlock +if (x > 0) { + print("positive"); +} else { + print("non-positive"); +} + +for (let i = 0; i < 10; i++) { + print(i); +} + +for (item in [1, 2, 3]) { + print(item); +} +``` + +**Functions:** +```hemlock +fn add(a: i32, b: i32): i32 { + return a + b; +} + +let double = fn(x) => x * 2; +``` + +### Exercises +- [ ] Write a function that calculates factorial recursively +- [ ] Create an array and find its maximum value +- [ ] Build an object representing a person with name and age + +### Resources +- [Tutorial](../getting-started/tutorial.md) +- [Syntax Guide](../language-guide/syntax.md) + +--- + +## Module 2: Understanding Memory + +**Duration:** 4-6 hours (core module) + +### Learning Objectives +- What memory is and how it's organized +- Stack vs heap allocation +- Pointers as addresses +- Why memory management matters + +### Content + +**Read First:** +- [Why Manual Memory Matters](./why-memory-matters.md) + +**Key Concepts:** +1. Memory is a giant array of bytes +2. Every byte has an address (number) +3. A pointer is just an address +4. Stack: automatic, fast, limited +5. Heap: manual, flexible, unlimited + +**Core Functions:** +```hemlock +alloc(n) // Request n bytes from heap +free(p) // Return memory to heap +ptr_read_i32(p) // Read 4 bytes as i32 +ptr_write_i32(p, v) // Write i32 to address +sizeof(type) // Size of type in bytes +memset(p, val, n) // Set n bytes to val +memcpy(dst, src, n) // Copy n bytes +``` + +### Exercises (Complete All) +- [ ] [Exercise 1: Your First Allocation](./exercises/01-first-allocation.md) +- [ ] [Exercise 2: Stack vs Heap](./exercises/02-stack-vs-heap.md) +- [ ] [Exercise 3: Pointers Explained](./exercises/03-pointers-explained.md) + +### Milestone Project +**Build a Ring Buffer:** A fixed-size circular queue that overwrites old data when full. +- Allocate buffer on heap +- Track head and tail pointers +- Implement push and pop operations +- Free buffer when done + +--- + +## Module 3: Memory Safety Patterns + +**Duration:** 3-4 hours + +### Learning Objectives +- Recognize common memory bugs +- Use `defer` for automatic cleanup +- Develop defensive coding habits +- Debug memory issues + +### Content + +**Read First:** +- [Common Mistakes and Debugging](./common-mistakes.md) + +**Key Patterns:** + +1. **Defer for Cleanup:** + ```hemlock + fn safe_processing() { + let p = alloc(100); + defer free(p); // Guaranteed cleanup + // ... work ... + } // defer runs here + ``` + +2. **Null After Free:** + ```hemlock + free(p); + p = ptr_null(); // Prevent use-after-free + ``` + +3. **Check Before Dereference:** + ```hemlock + if (p != ptr_null()) { + let value = ptr_read_i32(p); + } + ``` + +### Exercises +- [ ] Find and fix the bugs in a provided program +- [ ] Add defensive checks to your ring buffer +- [ ] Implement a simple memory leak detector + +### Resources +- [Memory Guide](../language-guide/memory.md) + +--- + +## Module 4: Types in Depth + +**Duration:** 2-3 hours + +### Learning Objectives +- Numeric type ranges and sizes +- Type promotion and coercion +- String internals (UTF-8, bytes vs characters) +- Custom types with `define` + +### Content + +**Numeric Types:** +| Type | Size | Range | +|------|------|-------| +| i8 | 1B | -128 to 127 | +| i16 | 2B | -32,768 to 32,767 | +| i32 | 4B | ±2 billion | +| i64 | 8B | ±9 quintillion | +| u8 | 1B | 0 to 255 | +| f32 | 4B | ~7 decimal digits | +| f64 | 8B | ~15 decimal digits | + +**String Internals:** +```hemlock +let s = "hello 🚀"; +print(len(s)); // 10 (bytes) +print(s.length); // 7 (characters) +print(s.byte_at(6)); // First byte of emoji +``` + +**Custom Types:** +```hemlock +define Point { + x: f64, + y: f64 +} + +define HasName { + name: string, + fn greet(): string +} +``` + +### Exercises +- [ ] Explore integer overflow behavior +- [ ] Write a function that counts emoji in a string +- [ ] Create a compound type for a game character + +--- + +## Module 5: Concurrency Fundamentals + +**Duration:** 4-5 hours + +### Learning Objectives +- Tasks and real parallelism +- Race conditions and why they happen +- Atomic operations +- Channels for communication + +### Content + +**Run First:** +- [examples/educational/concurrency_basics.hml](../../examples/educational/concurrency_basics.hml) + +**Key Concepts:** +1. `spawn` creates real OS threads +2. Shared mutable state causes race conditions +3. Atomics provide lock-free synchronization +4. Channels allow safe message passing + +**Core Functions:** +```hemlock +spawn(fn, args...) // Start parallel task +join(task) // Wait for result +detach(task) // Fire and forget +channel(size) // Create channel +ch.send(value) // Send message +ch.recv() // Receive message +atomic_add_i32(p, n) // Atomic addition +``` + +### Exercises +- [ ] Write a parallel map function +- [ ] Implement a thread-safe counter with atomics +- [ ] Build a producer-consumer pipeline with channels + +### Resources +- [Async/Concurrency Guide](../advanced/async-concurrency.md) +- [Atomics Reference](../advanced/atomics.md) + +--- + +## Module 6: Error Handling + +**Duration:** 2-3 hours + +### Learning Objectives +- try/catch/finally for recoverable errors +- panic for unrecoverable errors +- Error propagation patterns +- Using defer with errors + +### Content + +```hemlock +try { + let result = risky_operation(); +} catch (e) { + print("Caught error: " + e); +} finally { + cleanup(); // Always runs +} + +// For unrecoverable errors +if (critical_failure) { + panic("Cannot continue: " + reason); +} +``` + +**Pattern - Defer with Errors:** +```hemlock +fn process_file(path: string) { + let f = open(path, "r"); + defer f.close(); // Runs even if exception thrown + + try { + let data = f.read(); + process(data); + } catch (e) { + print("Error processing: " + e); + } +} // f.close() called here +``` + +### Exercises +- [ ] Write a function that validates input and throws on error +- [ ] Implement retry logic for flaky operations +- [ ] Create a RAII-style resource wrapper + +--- + +## Module 7: File I/O and System Interaction + +**Duration:** 2-3 hours + +### Learning Objectives +- Reading and writing files +- Command-line arguments +- Environment variables +- Running shell commands + +### Content + +**File I/O:** +```hemlock +let f = open("data.txt", "r"); +let content = f.read(); +f.close(); + +// With defer +let f = open("output.txt", "w"); +defer f.close(); +f.write("Hello, file!"); +``` + +**Command-Line Args:** +```hemlock +import { args } from "@stdlib/args"; + +let argv = args(); +print("Program: " + argv[0]); +print("Arg 1: " + argv[1]); +``` + +### Exercises +- [ ] Write a file copy utility +- [ ] Create a simple grep-like search tool +- [ ] Build a command-line calculator + +### Resources +- [File I/O Guide](../advanced/file-io.md) +- [Command Line Args](../advanced/command-line-args.md) + +--- + +## Module 8: FFI (Foreign Function Interface) + +**Duration:** 3-4 hours + +### Learning Objectives +- Declare and call C functions +- Work with C strings and pointers +- Load shared libraries dynamically +- Safety considerations with FFI + +### Content + +```hemlock +// Static binding +import "libc.so.6"; + +extern fn strlen(s: string): i32; +extern fn getpid(): i32; + +print("Length: " + strlen("Hello")); +print("PID: " + getpid()); + +// Dynamic binding +let lib = ffi_open("libm.so.6"); +let sqrt = ffi_bind(lib, "sqrt", [FFI_DOUBLE], FFI_DOUBLE); +print(sqrt(2.0)); +ffi_close(lib); +``` + +### Exercises +- [ ] Call math functions from libm +- [ ] Create a wrapper for a simple C library +- [ ] Implement a function that calls getenv/setenv + +### Resources +- [FFI Guide](../advanced/ffi.md) + +--- + +## Module 9: Building Real Programs + +**Duration:** Ongoing + +### Capstone Projects + +Choose one or more to solidify your learning: + +**1. Memory Allocator (Intermediate)** +Build a simple allocator on top of a fixed buffer. +- Implement `my_alloc` and `my_free` +- Track free blocks in a list +- Handle fragmentation + +**2. Concurrent Web Scraper (Intermediate)** +Fetch multiple URLs in parallel. +- Use `spawn` for parallel requests +- Collect results with channels +- Handle errors gracefully + +**3. Key-Value Store (Advanced)** +Build an in-memory database. +- Hash table implementation +- Thread-safe access with atomics +- Persistence to disk + +**4. Simple Shell (Advanced)** +Build a basic command-line shell. +- Parse commands +- Fork and execute processes +- Handle signals (Ctrl+C) + +--- + +## Assessment Checkpoints + +### After Module 2 (Memory Basics) +You should be able to: +- [ ] Explain the difference between stack and heap +- [ ] Allocate memory, write to it, read from it, free it +- [ ] Identify what a dangling pointer is +- [ ] Draw a memory diagram showing pointer relationships + +### After Module 5 (Concurrency) +You should be able to: +- [ ] Explain why race conditions happen +- [ ] Use atomics to prevent data races +- [ ] Implement a producer-consumer with channels +- [ ] Identify memory ownership issues with spawned tasks + +### After Module 8 (FFI) +You should be able to: +- [ ] Call C standard library functions from Hemlock +- [ ] Understand memory layout for FFI compatibility +- [ ] Wrap a simple C library for use in Hemlock + +--- + +## Quick Reference + +### Memory Functions +```hemlock +alloc(n) // Allocate n bytes +free(p) // Free allocation +buffer(n) // Bounds-checked buffer +talloc(type, n) // Typed allocation +memset(p, v, n) // Fill memory +memcpy(d, s, n) // Copy memory +``` + +### Pointer Operations +```hemlock +ptr_null() // Null pointer +&variable // Address of variable +ptr_read_*type*(p) // Read typed value +ptr_write_*type*(p, v) // Write typed value +``` + +### Concurrency +```hemlock +spawn(fn, args) // Start task +join(task) // Wait for result +channel(size) // Create channel +ch.send(v) // Send message +ch.recv() // Receive message +atomic_* // Atomic operations +``` + +--- + +## Getting Help + +- Read error messages carefully +- Check [Common Mistakes](./common-mistakes.md) +- Print intermediate values +- Use `hemlockc --keep-c` to see generated code +- Ask questions at https://github.com/hemlang/hemlock/issues + +--- + +*"The journey of a thousand miles begins with a single allocation."* diff --git a/docs/education/exercises/01-first-allocation.md b/docs/education/exercises/01-first-allocation.md new file mode 100644 index 00000000..47f53483 --- /dev/null +++ b/docs/education/exercises/01-first-allocation.md @@ -0,0 +1,306 @@ +# Exercise 1: Your First Allocation + +**Goal:** Understand the basics of manual memory allocation and deallocation. + +**Prerequisites:** None - this is where you start! + +--- + +## Concepts Introduced + +- `alloc(size)` - Request bytes from the heap +- `free(ptr)` - Return memory to the heap +- `ptr_write_*` / `ptr_read_*` - Write/read values through pointers +- Memory as a sequence of bytes + +--- + +## Part 1: Allocating a Single Integer + +An `i32` (32-bit integer) requires 4 bytes of storage. Let's allocate space for one: + +```hemlock +// Step 1: Request 4 bytes from the heap +let p = alloc(4); +print("Allocated memory at: " + p); + +// Step 2: Write a value to that memory +ptr_write_i32(p, 42); +print("Wrote value 42"); + +// Step 3: Read the value back +let value = ptr_read_i32(p); +print("Read back: " + value); + +// Step 4: Release the memory +free(p); +print("Memory freed"); +``` + +**Run this and observe:** +- The pointer value (memory address) +- The value survives write → read +- No errors on free + +--- + +## Part 2: Understanding Bytes + +Let's see how an integer is actually stored: + +```hemlock +let p = alloc(4); +ptr_write_i32(p, 0x12345678); // Hex value for clarity + +// Read individual bytes +print("Byte 0: " + ptr_read_u8(p)); +print("Byte 1: " + ptr_read_u8(p + 1)); +print("Byte 2: " + ptr_read_u8(p + 2)); +print("Byte 3: " + ptr_read_u8(p + 3)); + +free(p); +``` + +**Expected output (on little-endian systems):** +``` +Byte 0: 120 (0x78) +Byte 1: 86 (0x56) +Byte 2: 52 (0x34) +Byte 3: 18 (0x12) +``` + +**Key insight:** The bytes are stored in reverse order! This is called "little-endian" byte order, used by most modern CPUs (x86, ARM). The "least significant byte" comes first. + +--- + +## Part 3: Multiple Values + +Allocate space for multiple integers: + +```hemlock +// 3 integers × 4 bytes each = 12 bytes +let p = alloc(12); + +// Write three values +ptr_write_i32(p, 10); // First integer at offset 0 +ptr_write_i32(p + 4, 20); // Second integer at offset 4 +ptr_write_i32(p + 8, 30); // Third integer at offset 8 + +// Read them back +print("Value 0: " + ptr_read_i32(p)); +print("Value 1: " + ptr_read_i32(p + 4)); +print("Value 2: " + ptr_read_i32(p + 8)); + +free(p); +``` + +**Key insight:** Pointer arithmetic! `p + 4` means "4 bytes after p". You must calculate offsets manually. + +--- + +## Part 4: Using `sizeof` + +Instead of hardcoding sizes, use `sizeof`: + +```hemlock +let int_size = sizeof(i32); +print("Size of i32: " + int_size + " bytes"); + +let count = 5; +let p = alloc(count * int_size); + +for (let i = 0; i < count; i++) { + ptr_write_i32(p + i * int_size, i * 10); +} + +for (let i = 0; i < count; i++) { + print("arr[" + i + "] = " + ptr_read_i32(p + i * int_size)); +} + +free(p); +``` + +**Output:** +``` +Size of i32: 4 bytes +arr[0] = 0 +arr[1] = 10 +arr[2] = 20 +arr[3] = 30 +arr[4] = 40 +``` + +--- + +## Part 5: Using `memset` + +Initialize memory to a specific byte value: + +```hemlock +let p = alloc(16); + +// Fill with zeros +memset(p, 0, 16); +print("After memset(0): " + ptr_read_i32(p)); // 0 + +// Fill with a pattern (careful: this sets BYTES, not integers) +memset(p, 0xFF, 16); +print("After memset(0xFF): " + ptr_read_i32(p)); // -1 (all bits set) + +free(p); +``` + +**Key insight:** `memset` sets individual bytes, not values. `memset(p, 1, 4)` doesn't set an integer to 1 - it sets each byte to 1, resulting in `0x01010101` (16843009 in decimal). + +--- + +## Challenge Exercises + +### Challenge 1: Build an Array + +Write a program that: +1. Allocates space for 10 integers +2. Fills them with values 1-10 +3. Calculates and prints the sum +4. Frees the memory + +
+Solution + +```hemlock +let count = 10; +let p = alloc(count * sizeof(i32)); + +// Fill with 1-10 +for (let i = 0; i < count; i++) { + ptr_write_i32(p + i * sizeof(i32), i + 1); +} + +// Calculate sum +let sum = 0; +for (let i = 0; i < count; i++) { + sum = sum + ptr_read_i32(p + i * sizeof(i32)); +} + +print("Sum: " + sum); // Should be 55 + +free(p); +``` +
+ +### Challenge 2: Swap Two Values + +Write a function that swaps two integers in memory using a temporary variable: + +```hemlock +fn swap(p: ptr, i: i32, j: i32) { + // Your code here + // Swap values at offsets i*4 and j*4 +} + +let p = alloc(12); +ptr_write_i32(p, 10); +ptr_write_i32(p + 4, 20); +ptr_write_i32(p + 8, 30); + +swap(p, 0, 2); // Swap first and third + +print(ptr_read_i32(p)); // Should be 30 +print(ptr_read_i32(p + 8)); // Should be 10 + +free(p); +``` + +
+Solution + +```hemlock +fn swap(p: ptr, i: i32, j: i32) { + let offset_i = i * sizeof(i32); + let offset_j = j * sizeof(i32); + + let temp = ptr_read_i32(p + offset_i); + ptr_write_i32(p + offset_i, ptr_read_i32(p + offset_j)); + ptr_write_i32(p + offset_j, temp); +} +``` +
+ +### Challenge 3: Different Types + +Allocate memory for a "struct" containing: +- An i32 (4 bytes) +- An f64 (8 bytes) +- An i8 (1 byte) + +Write values and read them back. + +
+Solution + +```hemlock +// Layout: i32 at 0, f64 at 4, i8 at 12 +// Total size: 13 bytes (but we'd typically align to 16) +let p = alloc(16); + +ptr_write_i32(p, 42); // i32 at offset 0 +ptr_write_f64(p + 4, 3.14159); // f64 at offset 4 +ptr_write_i8(p + 12, 7); // i8 at offset 12 + +print("i32: " + ptr_read_i32(p)); +print("f64: " + ptr_read_f64(p + 4)); +print("i8: " + ptr_read_i8(p + 12)); + +free(p); +``` + +**Note:** Real C compilers add padding for alignment. Hemlock's interpreter is more forgiving, but understanding alignment matters for FFI. +
+ +--- + +## Common Mistakes + +### Mistake 1: Wrong Size + +```hemlock +let p = alloc(4); // Only 4 bytes +ptr_write_i64(p, 123); // i64 needs 8 bytes - overflow! +free(p); +``` + +### Mistake 2: Off-by-One in Offset + +```hemlock +let p = alloc(12); +ptr_write_i32(p, 1); +ptr_write_i32(p + 3, 2); // WRONG! Should be p + 4 + // This overlaps with first integer +free(p); +``` + +### Mistake 3: Forgetting to Free + +```hemlock +fn leaky() { + let p = alloc(100); + // ... use p ... + // Forgot free(p)! +} +``` + +--- + +## Key Takeaways + +1. **`alloc(n)`** requests `n` bytes from the heap and returns a pointer +2. **Pointer arithmetic** is in bytes, not elements +3. **`sizeof(type)`** gives the size of a type in bytes +4. **Always `free()`** what you `alloc()` +5. **Memory is just bytes** - interpretation depends on how you read it + +--- + +## Next Exercise + +Continue to [Exercise 2: Stack vs Heap](./02-stack-vs-heap.md) to understand where your variables actually live. diff --git a/docs/education/exercises/02-stack-vs-heap.md b/docs/education/exercises/02-stack-vs-heap.md new file mode 100644 index 00000000..b2bfb3c9 --- /dev/null +++ b/docs/education/exercises/02-stack-vs-heap.md @@ -0,0 +1,434 @@ +# Exercise 2: Stack vs Heap + +**Goal:** Understand the difference between stack and heap memory, and when to use each. + +**Prerequisites:** [Exercise 1: Your First Allocation](./01-first-allocation.md) + +--- + +## Concepts Introduced + +- Stack memory: automatic, fast, limited +- Heap memory: manual, flexible, unlimited +- Variable lifetimes +- Dangling pointers + +--- + +## Part 1: Stack Variables + +When you declare a variable in a function, it lives on the stack: + +```hemlock +fn stack_demo() { + let a = 10; // 'a' is on the stack + let b = 20; // 'b' is on the stack, right after 'a' + let c = 30; // 'c' follows 'b' + + print("a = " + a); + print("b = " + b); + print("c = " + c); + + // When this function returns, a, b, c are automatically gone +} + +stack_demo(); +// a, b, c no longer exist - their memory is reclaimed +``` + +**Key properties of stack:** +- **Automatic:** Variables are created when declared, destroyed when scope ends +- **Fast:** Allocation is just moving a pointer +- **LIFO:** Last In, First Out - like a stack of plates + +--- + +## Part 2: Heap Allocation + +Heap memory persists until you explicitly free it: + +```hemlock +fn heap_demo(): ptr { + let p = alloc(4); // Allocate on heap + ptr_write_i32(p, 42); + return p; // Pointer survives function return! +} + +let ptr = heap_demo(); +print("Value: " + ptr_read_i32(ptr)); // Still valid: 42 +free(ptr); // Must free manually +``` + +**Key properties of heap:** +- **Manual:** You control when memory is allocated and freed +- **Persistent:** Survives function returns +- **Flexible:** Allocate any size at runtime +- **Slower:** Allocator must search for free space + +--- + +## Part 3: The Dangling Pointer Problem + +What happens if you return a pointer to a stack variable? + +```hemlock +fn bad_idea(): ptr { + let x = 42; // x is on the stack + return &x; // Return address of x + // x dies here! +} + +// DON'T RUN THIS - it demonstrates undefined behavior +// let p = bad_idea(); +// print(ptr_read_i32(p)); // Reading garbage! +``` + +The pointer `&x` becomes "dangling" - it points to memory that no longer belongs to `x`. Reading it gives unpredictable results. + +**The fix: Use the heap when data must outlive the function:** + +```hemlock +fn good_idea(): ptr { + let p = alloc(4); // Heap allocation + ptr_write_i32(p, 42); + return p; // Caller owns this memory now +} + +let p = good_idea(); +print(ptr_read_i32(p)); // 42 - safe! +free(p); // Caller must free +``` + +--- + +## Part 4: Visualizing Memory + +Let's trace what happens during execution: + +```hemlock +fn outer() { + let a = 1; // Stack: [a=1] + let p = alloc(4); // Stack: [a=1, p=0x1000], Heap: [4 bytes at 0x1000] + ptr_write_i32(p, 100); // Heap now contains 100 + + inner(p); // Pass pointer to inner + + print(ptr_read_i32(p)); // 200 - inner modified it! + free(p); +} + +fn inner(p: ptr) { + let b = 2; // Stack: [a=1, p=0x1000, b=2] + ptr_write_i32(p, 200); // Modify heap through pointer + // b dies here +} + +outer(); +// a, p die here, heap is freed +``` + +**Memory timeline:** +``` +Time 1 (enter outer): Stack: [a=1] +Time 2 (after alloc): Stack: [a=1, p→heap], Heap: [_] +Time 3 (after write): Stack: [a=1, p→heap], Heap: [100] +Time 4 (enter inner): Stack: [a=1, p→heap, b=2], Heap: [100] +Time 5 (inner writes): Stack: [a=1, p→heap, b=2], Heap: [200] +Time 6 (exit inner): Stack: [a=1, p→heap], Heap: [200] +Time 7 (exit outer): Stack: [], Heap: [] (freed) +``` + +--- + +## Part 5: Stack Size Limits + +The stack is limited (typically 8MB). Large allocations should use the heap: + +```hemlock +// BAD: This might overflow the stack +fn stack_overflow() { + let arr = [0; 10000000]; // 10 million elements on stack? Dangerous! +} + +// GOOD: Large data on heap +fn safe_large_data() { + let size = 10000000 * sizeof(i32); + let p = alloc(size); + // ... use p ... + free(p); +} +``` + +--- + +## Part 6: Lifetime Patterns + +### Pattern 1: Create and Destroy in Same Scope + +```hemlock +fn process() { + let p = alloc(100); + // ... use p ... + free(p); // Clean up before returning +} +``` + +**Best for:** Temporary working memory + +### Pattern 2: Factory Function (Caller Owns) + +```hemlock +fn create_thing(): ptr { + let p = alloc(64); + // ... initialize p ... + return p; // Caller must free +} + +let thing = create_thing(); +// ... use thing ... +free(thing); +``` + +**Best for:** Creating objects with longer lifetimes + +### Pattern 3: Pass Ownership + +```hemlock +fn consume_thing(p: ptr) { + // ... use p ... + free(p); // This function takes ownership and frees +} + +let p = alloc(64); +consume_thing(p); +// p is now invalid - don't use it! +``` + +**Best for:** Transfer of responsibility + +### Pattern 4: Borrow (No Ownership Transfer) + +```hemlock +fn inspect_thing(p: ptr) { + // Read from p, but don't free + print(ptr_read_i32(p)); +} + +let p = alloc(64); +ptr_write_i32(p, 42); +inspect_thing(p); // Borrows p +inspect_thing(p); // Can borrow again +free(p); // Original owner frees +``` + +**Best for:** Read-only or temporary access + +--- + +## Challenge Exercises + +### Challenge 1: Identify the Bug + +What's wrong with this code? + +```hemlock +fn make_array(): ptr { + let arr = [1, 2, 3, 4, 5]; + return &arr; // ??? +} + +let p = make_array(); +// What happens here? +``` + +
+Answer + +`arr` is a managed array on the stack. Returning `&arr` creates a dangling pointer because `arr` is destroyed when the function returns. The correct approach is to either return the array directly (Hemlock arrays are reference-counted) or allocate on the heap: + +```hemlock +// Option 1: Return managed array (correct) +fn make_array(): array { + return [1, 2, 3, 4, 5]; +} + +// Option 2: Manual heap allocation (correct) +fn make_array_manual(): ptr { + let p = alloc(5 * sizeof(i32)); + for (let i = 0; i < 5; i++) { + ptr_write_i32(p + i * sizeof(i32), i + 1); + } + return p; // Caller frees +} +``` +
+ +### Challenge 2: Memory Tracker + +Write a program that tracks allocations and frees to detect leaks: + +```hemlock +let allocated = 0; +let freed = 0; + +fn tracked_alloc(size: i32): ptr { + // Your code: increment allocated, call alloc +} + +fn tracked_free(p: ptr) { + // Your code: increment freed, call free +} + +// Test it +let p1 = tracked_alloc(100); +let p2 = tracked_alloc(200); +tracked_free(p1); +// Forgot to free p2! + +print("Allocated: " + allocated); +print("Freed: " + freed); +print("Leaked: " + (allocated - freed)); +``` + +
+Solution + +```hemlock +let allocated = 0; +let freed = 0; + +fn tracked_alloc(size: i32): ptr { + allocated = allocated + 1; + return alloc(size); +} + +fn tracked_free(p: ptr) { + freed = freed + 1; + free(p); +} + +let p1 = tracked_alloc(100); +let p2 = tracked_alloc(200); +tracked_free(p1); + +print("Allocated: " + allocated); // 2 +print("Freed: " + freed); // 1 +print("Leaked: " + (allocated - freed)); // 1 +``` +
+ +### Challenge 3: Implement a Resizable Buffer + +Create a buffer that can grow: + +```hemlock +// Start with 10 elements capacity +let capacity = 10; +let size = 0; +let data = alloc(capacity * sizeof(i32)); + +fn push(value: i32) { + // Your code: + // 1. If size == capacity, grow the buffer + // 2. Write value at position 'size' + // 3. Increment size +} + +fn get(index: i32): i32 { + // Your code: return value at index +} + +// Test +for (let i = 0; i < 25; i++) { + push(i * 10); +} + +for (let i = 0; i < 25; i++) { + print("data[" + i + "] = " + get(i)); +} + +free(data); +``` + +
+Solution + +```hemlock +let capacity = 10; +let size = 0; +let data = alloc(capacity * sizeof(i32)); + +fn push(value: i32) { + if (size == capacity) { + // Double the capacity + let new_capacity = capacity * 2; + let new_data = alloc(new_capacity * sizeof(i32)); + + // Copy old data + memcpy(new_data, data, size * sizeof(i32)); + + // Free old buffer + free(data); + + // Update globals + data = new_data; + capacity = new_capacity; + print("Grew to capacity: " + capacity); + } + + ptr_write_i32(data + size * sizeof(i32), value); + size = size + 1; +} + +fn get(index: i32): i32 { + if (index < 0 || index >= size) { + panic("Index out of bounds"); + } + return ptr_read_i32(data + index * sizeof(i32)); +} +``` +
+ +--- + +## Key Takeaways + +1. **Stack:** Fast, automatic, limited size, LIFO, dies with scope +2. **Heap:** Manual, flexible, unlimited (by RAM), persists until freed +3. **Never return pointers to stack variables** - they become dangling +4. **Document ownership** - who allocates, who frees? +5. **Large data belongs on the heap** - don't overflow the stack + +--- + +## Memory Layout Diagram + +``` +┌──────────────────────────────────────────────┐ +│ STACK │ +│ (grows downward) │ +│ ┌────────────────────────────────────────┐ │ +│ │ main() locals │ │ +│ │ a: i32 = 10 │ │ +│ │ p: ptr = 0x7fa0 │←─┘ +│ ├────────────────────────────────────────┤ +│ │ inner() locals │ +│ │ b: i32 = 20 │ +│ └────────────────────────────────────────┘ +│ ↓ │ +│ (empty space) │ +│ ↑ │ +│ ┌────────────────────────────────────────┐ │ +│ │ 0x7fa0: [100 bytes allocated] │←─┐ +│ │ 0x8000: [64 bytes allocated] │ │ +│ └────────────────────────────────────────┘ │ +│ HEAP │ +│ (grows upward) │ +└──────────────────────────────────────────────┘ +``` + +--- + +## Next Exercise + +Continue to [Exercise 3: Pointers Explained](./03-pointers-explained.md) to demystify pointer arithmetic. diff --git a/docs/education/exercises/03-pointers-explained.md b/docs/education/exercises/03-pointers-explained.md new file mode 100644 index 00000000..27fb5ff8 --- /dev/null +++ b/docs/education/exercises/03-pointers-explained.md @@ -0,0 +1,396 @@ +# Exercise 3: Pointers Explained + +**Goal:** Demystify pointers - they're just numbers that happen to be addresses. + +**Prerequisites:** [Exercise 2: Stack vs Heap](./02-stack-vs-heap.md) + +--- + +## Concepts Introduced + +- Pointers as memory addresses (numbers) +- Address-of operator (`&`) +- Pointer arithmetic +- Null pointers +- Multi-level pointers (pointers to pointers) + +--- + +## The Big Reveal: Pointers Are Just Numbers + +A pointer is simply an integer that represents a memory address. That's it. + +```hemlock +let p = alloc(4); +print(p); // ptr(0x7f8a2b3c4d50) - it's just a number! +print(typeof(p)); // "ptr" + +// The number tells you WHERE in memory your data lives +// Address 0x7f8a2b3c4d50 is byte 140,234,567,890,000 in memory +``` + +**Mental model:** Memory is like a giant hotel. Each room has a number (address). A pointer is like a room key card - it just holds the room number. + +--- + +## Part 1: The Address-Of Operator + +`&variable` gives you the address where a variable is stored: + +```hemlock +let x = 42; +print("x = " + x); // 42 +print("address of x: " + &x); // Some address like ptr(0x7ffc...) + +// You can read through the address +let p = &x; +print("value at p: " + ptr_read_i32(p)); // 42 +``` + +This works for stack variables and heap allocations alike. + +--- + +## Part 2: Pointer Arithmetic + +Since pointers are numbers, you can do math on them: + +```hemlock +let p = alloc(20); // 20 bytes = space for 5 integers + +// Write values at different offsets +ptr_write_i32(p, 100); // Address p + 0 +ptr_write_i32(p + 4, 200); // Address p + 4 +ptr_write_i32(p + 8, 300); // Address p + 8 +ptr_write_i32(p + 12, 400); // Address p + 12 +ptr_write_i32(p + 16, 500); // Address p + 16 + +// Read them back +for (let i = 0; i < 5; i++) { + let offset = i * 4; // 4 bytes per integer + print("p[" + i + "] at offset " + offset + " = " + ptr_read_i32(p + offset)); +} + +free(p); +``` + +**Key insight:** `p + 4` means "the address 4 bytes after p". Hemlock does byte-level arithmetic (unlike C, which scales by type size). + +--- + +## Part 3: Why Pointer Arithmetic is Byte-Based + +Consider this memory layout: + +``` +Address: 1000 1001 1002 1003 1004 1005 1006 1007 + ├──────────────────┤├──────────────────┤ + │ i32 = 42 ││ i32 = 99 │ + └──────────────────┘└──────────────────┘ +``` + +To move from one i32 to the next, you add 4 (bytes), not 1: + +```hemlock +let p = alloc(8); +ptr_write_i32(p, 42); +ptr_write_i32(p + 4, 99); // Next i32 is 4 bytes later + +// WRONG: p + 1 would point into the MIDDLE of the first integer! +// ptr_write_i32(p + 1, 99); // This corrupts data! + +free(p); +``` + +--- + +## Part 4: Walking Through an Array + +```hemlock +// Create an "array" of 10 doubles +let count = 10; +let elem_size = sizeof(f64); // 8 bytes +let p = alloc(count * elem_size); + +// Fill with squares +for (let i = 0; i < count; i++) { + let addr = p + i * elem_size; + ptr_write_f64(addr, (i + 1) * (i + 1)); +} + +// Sum them up +let sum = 0.0; +let current = p; +for (let i = 0; i < count; i++) { + sum = sum + ptr_read_f64(current); + current = current + elem_size; // Move to next element +} + +print("Sum of squares 1-10: " + sum); // 385.0 + +free(p); +``` + +--- + +## Part 5: Null Pointers + +A null pointer is address 0 - a "nowhere" address: + +```hemlock +let p = ptr_null(); +print(p); // ptr(0x0) or ptr(null) +print(p == ptr_null()); // true + +// Dereferencing null is UNDEFINED BEHAVIOR +// ptr_read_i32(p); // DON'T DO THIS - crash or worse! + +// Always check before dereferencing +fn safe_read(p: ptr): i32 { + if (p == ptr_null()) { + print("Error: null pointer"); + return 0; + } + return ptr_read_i32(p); +} +``` + +**Convention:** Use `ptr_null()` to represent "no value" or "not initialized". + +--- + +## Part 6: Pointers to Pointers + +A pointer can hold the address of another pointer: + +```hemlock +let x = 42; +let p1 = &x; // p1 points to x +let p2 = &p1; // p2 points to p1 (pointer to pointer) + +print("x = " + x); +print("*p1 = " + ptr_read_i32(p1)); // 42 +print("**p2 = " + ptr_read_i32(ptr_read_ptr(p2))); // 42 + +// Modify x through p2 +let inner_ptr = ptr_read_ptr(p2); // Get p1's value (address of x) +ptr_write_i32(inner_ptr, 100); // Write to x through p1 +print("x is now: " + x); // 100 +``` + +**Why useful?** Pointers to pointers are used for: +- Arrays of strings (each string is a pointer) +- Linked data structures +- Output parameters (modify the caller's pointer) + +--- + +## Part 7: The `memcpy` Function + +Copy bytes from one location to another: + +```hemlock +let src = alloc(20); +let dst = alloc(20); + +// Initialize source +for (let i = 0; i < 5; i++) { + ptr_write_i32(src + i * 4, (i + 1) * 10); +} + +// Copy all 20 bytes +memcpy(dst, src, 20); + +// Verify +print("Source:"); +for (let i = 0; i < 5; i++) { + print(" src[" + i + "] = " + ptr_read_i32(src + i * 4)); +} + +print("Destination (after copy):"); +for (let i = 0; i < 5; i++) { + print(" dst[" + i + "] = " + ptr_read_i32(dst + i * 4)); +} + +free(src); +free(dst); +``` + +--- + +## Challenge Exercises + +### Challenge 1: Implement `memcpy` Yourself + +Write a function that copies bytes from source to destination: + +```hemlock +fn my_memcpy(dst: ptr, src: ptr, n: i32) { + // Copy n bytes from src to dst + // Hint: Use ptr_read_u8 and ptr_write_u8 for byte-by-byte copy +} + +// Test it +let src = alloc(16); +let dst = alloc(16); +ptr_write_i32(src, 0x12345678); +ptr_write_i32(src + 4, 0xDEADBEEF); + +my_memcpy(dst, src, 8); +print(ptr_read_i32(dst)); // Should match src +print(ptr_read_i32(dst + 4)); // Should match src + +free(src); +free(dst); +``` + +
+Solution + +```hemlock +fn my_memcpy(dst: ptr, src: ptr, n: i32) { + for (let i = 0; i < n; i++) { + let byte = ptr_read_u8(src + i); + ptr_write_u8(dst + i, byte); + } +} +``` +
+ +### Challenge 2: Reverse an Array In-Place + +Given a pointer to an array of integers, reverse it without allocating new memory: + +```hemlock +fn reverse_array(p: ptr, count: i32) { + // Swap elements from both ends, moving toward middle +} + +let p = alloc(5 * sizeof(i32)); +for (let i = 0; i < 5; i++) { + ptr_write_i32(p + i * sizeof(i32), i + 1); +} + +print("Before: "); +// Should print: 1 2 3 4 5 + +reverse_array(p, 5); + +print("After: "); +// Should print: 5 4 3 2 1 + +free(p); +``` + +
+Solution + +```hemlock +fn reverse_array(p: ptr, count: i32) { + let elem_size = sizeof(i32); + let left = 0; + let right = count - 1; + + while (left < right) { + let left_addr = p + left * elem_size; + let right_addr = p + right * elem_size; + + // Swap + let temp = ptr_read_i32(left_addr); + ptr_write_i32(left_addr, ptr_read_i32(right_addr)); + ptr_write_i32(right_addr, temp); + + left = left + 1; + right = right - 1; + } +} +``` +
+ +### Challenge 3: Build a Linked List Node + +Create a simple linked list structure using raw pointers: + +```hemlock +// Node layout: +// Bytes 0-3: value (i32) +// Bytes 4-11: next pointer (ptr) + +fn create_node(value: i32, next: ptr): ptr { + // Allocate and initialize a node +} + +fn get_value(node: ptr): i32 { + // Return the value from a node +} + +fn get_next(node: ptr): ptr { + // Return the next pointer from a node +} + +// Create list: 1 -> 2 -> 3 -> null +let n3 = create_node(3, ptr_null()); +let n2 = create_node(2, n3); +let n1 = create_node(1, n2); + +// Walk the list +let current = n1; +while (current != ptr_null()) { + print(get_value(current)); + current = get_next(current); +} + +// Free all nodes +free(n1); +free(n2); +free(n3); +``` + +
+Solution + +```hemlock +fn create_node(value: i32, next: ptr): ptr { + let node = alloc(12); // 4 bytes for i32 + 8 bytes for ptr + ptr_write_i32(node, value); + ptr_write_ptr(node + 4, next); + return node; +} + +fn get_value(node: ptr): i32 { + return ptr_read_i32(node); +} + +fn get_next(node: ptr): ptr { + return ptr_read_ptr(node + 4); +} +``` +
+ +--- + +## Pointer Arithmetic Cheat Sheet + +| Operation | Meaning | +|-----------|---------| +| `p + n` | Address n bytes after p | +| `p - n` | Address n bytes before p | +| `p + i * sizeof(T)` | Address of i-th element of type T | +| `p == q` | Do p and q point to the same address? | +| `p == ptr_null()` | Is p a null pointer? | + +--- + +## Key Takeaways + +1. **Pointers are addresses** - Just numbers representing memory locations +2. **Arithmetic is byte-based** - Add `sizeof(T)` to move between elements +3. **Null is zero** - `ptr_null()` represents "no address" +4. **Pointers can point to pointers** - Multi-level indirection is possible +5. **Check before dereferencing** - Null dereference is undefined behavior + +--- + +## Next Exercise + +Continue to [Exercise 4: Numeric Types](./04-numeric-types.md) to understand how numbers are represented in memory. diff --git a/docs/education/from-high-level.md b/docs/education/from-high-level.md new file mode 100644 index 00000000..36b2d936 --- /dev/null +++ b/docs/education/from-high-level.md @@ -0,0 +1,721 @@ +# From Python/JavaScript to Systems Programming + +> "You already know how to program. Now learn what the computer is actually doing." + +This guide helps programmers from Python, JavaScript, Ruby, or other high-level languages transition to systems programming concepts using Hemlock. + +--- + +## What You Already Know (That Still Works) + +Good news! Most of your knowledge transfers directly: + +| Python/JS | Hemlock | +|-----------|---------| +| `x = 42` | `let x = 42;` | +| `if x > 0:` | `if (x > 0) { }` | +| `for i in range(10):` | `for (let i = 0; i < 10; i++) { }` | +| `def add(a, b):` | `fn add(a, b) { }` | +| `[1, 2, 3]` | `[1, 2, 3]` | +| `{"key": value}` | `{ key: value }` | +| `len(x)` | `len(x)` | +| `type(x)` | `typeof(x)` | + +**The basics are the same.** The differences are in what happens underneath. + +--- + +## The Five Big Mindset Shifts + +### 1. Memory Is Your Responsibility + +**In Python:** +```python +def process(): + data = [0] * 1000000 # Allocated somewhere + return data # Still exists +# Eventually garbage collected (when? who knows?) +``` + +**In Hemlock:** +```hemlock +fn process(): ptr { + let p = alloc(1000000); // YOU allocate + return p; // YOU track it +} + +let data = process(); +// ... use data ... +free(data); // YOU free it +``` + +**The shift:** You decide when memory is created and destroyed. No garbage collector. No "eventually." + +--- + +### 2. Types Have Sizes + +**In Python:** +```python +x = 42 # An "integer" - could be any size +y = 99999999999 # Still an "integer" - Python handles it +``` + +**In Hemlock:** +```hemlock +let x: i32 = 42; // 32 bits = 4 bytes, range: -2B to +2B +let y: i64 = 99999999999; // 64 bits = 8 bytes, larger range +let z: i8 = 127; // 8 bits = 1 byte, range: -128 to 127 +``` + +**The shift:** Types determine how many bytes a value occupies and what values it can hold. + +**Why it matters:** +```hemlock +let z: i8 = 127; +z = z + 1; // OVERFLOW: wraps to -128! + +let x: i32 = 127; +x = x + 1; // Fine: 128 +``` + +--- + +### 3. Variables Are Just Memory Locations + +**In Python:** +```python +x = [1, 2, 3] +y = x # y references the same list +y.append(4) +print(x) # [1, 2, 3, 4] - x changed too! +``` + +**In Hemlock with managed arrays (similar behavior):** +```hemlock +let x = [1, 2, 3]; +let y = x; // y references same array +y.push(4); +print(x); // [1, 2, 3, 4] +``` + +**In Hemlock with raw pointers (explicit control):** +```hemlock +let p = alloc(12); +ptr_write_i32(p, 1); +ptr_write_i32(p + 4, 2); +ptr_write_i32(p + 8, 3); + +let q = p; // q is the SAME address +ptr_write_i32(q, 99); +print(ptr_read_i32(p)); // 99 - same memory! + +// To copy, you must allocate new memory +let copy = alloc(12); +memcpy(copy, p, 12); +ptr_write_i32(copy, 0); +print(ptr_read_i32(p)); // Still 99 - copy is separate + +free(p); +free(copy); +``` + +**The shift:** Assignment copies the value (or reference), not the data. Understand what you're copying. + +--- + +### 4. Strings Are Byte Arrays + +**In Python:** +```python +s = "hello" +s[0] = "H" # ERROR: strings are immutable +s = "H" + s[1:] # Create new string +``` + +**In Hemlock:** +```hemlock +let s = "hello"; +s[0] = 'H'; // WORKS: strings are mutable +print(s); // "Hello" + +// Strings are UTF-8 byte sequences +print(len("hello")); // 5 bytes +print(len("🚀")); // 4 bytes (emoji is multi-byte) +print("🚀".length); // 1 character +``` + +**The shift:** Strings are sequences of bytes. Character count ≠ byte count for non-ASCII. + +--- + +### 5. Concurrency Is Parallel (Really) + +**In Python:** +```python +import threading +def work(): + # GIL means only one thread runs Python at a time + pass + +# "Threads" don't actually run in parallel for CPU work +``` + +**In Hemlock:** +```hemlock +async fn compute(): i32 { + // This ACTUALLY runs on another CPU core + let sum = 0; + for (let i = 0; i < 1000000; i++) { + sum = sum + i; + } + return sum; +} + +let t1 = spawn(compute); +let t2 = spawn(compute); // Both running simultaneously! + +let r1 = join(t1); +let r2 = join(t2); +``` + +**The shift:** Hemlock's `spawn` creates real OS threads with real parallelism. This means real race conditions if you're not careful. + +--- + +## Python to Hemlock Translation Guide + +### Variables + +```python +# Python +x = 42 +name = "Alice" +items = [1, 2, 3] +person = {"name": "Bob", "age": 30} +``` + +```hemlock +// Hemlock +let x = 42; +let name = "Alice"; +let items = [1, 2, 3]; +let person = { name: "Bob", age: 30 }; +``` + +### Functions + +```python +# Python +def greet(name, greeting="Hello"): + return f"{greeting}, {name}!" + +result = greet("Alice") +``` + +```hemlock +// Hemlock +fn greet(name: string, greeting?: "Hello"): string { + return `${greeting}, ${name}!`; +} + +let result = greet("Alice"); +``` + +### Control Flow + +```python +# Python +if x > 0: + print("positive") +elif x < 0: + print("negative") +else: + print("zero") + +for i in range(10): + print(i) + +while condition: + do_something() +``` + +```hemlock +// Hemlock +if (x > 0) { + print("positive"); +} else if (x < 0) { + print("negative"); +} else { + print("zero"); +} + +for (let i = 0; i < 10; i++) { + print(i); +} + +while (condition) { + do_something(); +} +``` + +### Lists/Arrays + +```python +# Python +items = [1, 2, 3] +items.append(4) +first = items[0] +length = len(items) +``` + +```hemlock +// Hemlock +let items = [1, 2, 3]; +items.push(4); +let first = items[0]; +let length = len(items); +``` + +### Dictionaries/Objects + +```python +# Python +person = {"name": "Alice", "age": 30} +name = person["name"] +person["email"] = "alice@example.com" +``` + +```hemlock +// Hemlock +let person = { name: "Alice", age: 30 }; +let name = person.name; +person.email = "alice@example.com"; +``` + +### Error Handling + +```python +# Python +try: + risky_operation() +except Exception as e: + print(f"Error: {e}") +finally: + cleanup() +``` + +```hemlock +// Hemlock +try { + risky_operation(); +} catch (e) { + print("Error: " + e); +} finally { + cleanup(); +} +``` + +### Classes vs Defines + +```python +# Python +class Person: + def __init__(self, name, age): + self.name = name + self.age = age + + def greet(self): + return f"Hi, I'm {self.name}" + +alice = Person("Alice", 30) +``` + +```hemlock +// Hemlock (using define + factory) +define Person { + name: string, + age: i32 +} + +fn create_person(name: string, age: i32): Person { + return { name: name, age: age }; +} + +fn greet(p: Person): string { + return `Hi, I'm ${p.name}`; +} + +let alice: Person = create_person("Alice", 30); +print(greet(alice)); +``` + +--- + +## JavaScript to Hemlock Translation Guide + +### Variables + +```javascript +// JavaScript +let x = 42; +const name = "Alice"; +var items = [1, 2, 3]; +``` + +```hemlock +// Hemlock (all are mutable, use const modifier for params) +let x = 42; +let name = "Alice"; +let items = [1, 2, 3]; +``` + +### Arrow Functions + +```javascript +// JavaScript +const double = x => x * 2; +const add = (a, b) => a + b; +``` + +```hemlock +// Hemlock +let double = fn(x) => x * 2; +let add = fn(a, b) => a + b; + +// Or with types +fn double(x: i32): i32 => x * 2; +``` + +### Async/Await + +```javascript +// JavaScript +async function fetchData() { + const response = await fetch(url); + return await response.json(); +} +``` + +```hemlock +// Hemlock +async fn fetch_data(): object { + let response = await http_get(url); + return response.body.deserialize(); +} + +let task = spawn(fetch_data); +let data = join(task); +``` + +### Promises vs Tasks + +```javascript +// JavaScript +const promise = doWork(); +promise.then(result => console.log(result)); +``` + +```hemlock +// Hemlock +let task = spawn(do_work); +let result = join(task); // Blocks until complete +print(result); +``` + +### Destructuring + +```javascript +// JavaScript +const { name, age } = person; +const [first, ...rest] = items; +``` + +```hemlock +// Hemlock (no destructuring yet - explicit access) +let name = person.name; +let age = person.age; +let first = items[0]; +let rest = items.slice(1); +``` + +### Spread Operator + +```javascript +// JavaScript +const combined = { ...defaults, ...options }; +const all = [...arr1, ...arr2]; +``` + +```hemlock +// Hemlock +let combined = { ...defaults, ...options }; // Object spread works! +let all = arr1.concat(arr2); // Use concat for arrays +``` + +### Null Coalescing + +```javascript +// JavaScript +const name = user?.name ?? "Anonymous"; +``` + +```hemlock +// Hemlock (same syntax!) +let name = user?.name ?? "Anonymous"; +``` + +--- + +## Things That Don't Exist (and Why) + +### No Garbage Collector + +**Why not?** Hemlock is for learning systems programming. GC hides the fundamental concept of memory management. + +**What to do instead:** +```hemlock +let p = alloc(100); +defer free(p); // Automatic cleanup at end of scope +``` + +### No Classes + +**Why not?** Hemlock uses composition over inheritance. `define` creates structural types. + +**What to do instead:** +```hemlock +define Shape { + fn area(): f64 +} + +let circle: Shape = { + radius: 5.0, + area: fn(): f64 => 3.14159 * self.radius * self.radius +}; +``` + +### No Automatic String Conversion + +**Why not?** Implicit conversions hide behavior. + +**What to do instead:** +```hemlock +let x = 42; +print("Value: " + x); // Explicit in template +// Or: +print(`Value: ${x}`); // Template string +``` + +### No `===` Operator + +**Why not?** Hemlock doesn't have JavaScript's type coercion issues. + +**What to do instead:** +```hemlock +// == does what you expect +42 == 42 // true +42 == "42" // false (different types) +``` + +--- + +## Memory Patterns for High-Level Programmers + +### Pattern 1: Use Defer for Cleanup + +```hemlock +fn process_file(path: string) { + let f = open(path, "r"); + defer f.close(); // Guaranteed to run + + let data = alloc(1000); + defer free(data); // Multiple defers OK, run in reverse order + + // ... do work ... + // Both cleanups happen automatically +} +``` + +### Pattern 2: Factory Functions Return Ownership + +```hemlock +// Caller must free the result +fn create_buffer(size: i32): ptr { + let p = alloc(size); + memset(p, 0, size); + return p; +} + +let buf = create_buffer(100); +// ... use buf ... +free(buf); // Caller's responsibility +``` + +### Pattern 3: Borrow for Read-Only Access + +```hemlock +// Function reads but doesn't own +fn sum_array(p: ptr, count: i32): i32 { + let total = 0; + for (let i = 0; i < count; i++) { + total = total + ptr_read_i32(p + i * sizeof(i32)); + } + return total; // Does NOT free p +} + +let data = alloc(20); +// ... fill data ... +let s = sum_array(data, 5); // Borrows data +free(data); // Original owner frees +``` + +### Pattern 4: Use Managed Types When Possible + +```hemlock +// For most code, managed arrays/objects are fine +let items = [1, 2, 3, 4, 5]; // No manual memory management +items.push(6); +items.pop(); +// Reference counted - cleaned up automatically + +// Only use raw pointers when you need: +// - Interop with C (FFI) +// - Custom memory layouts +// - Maximum control +``` + +--- + +## Common "Gotchas" for High-Level Programmers + +### 1. Semicolons Are Required + +```hemlock +// WRONG +let x = 42 +print(x) + +// RIGHT +let x = 42; +print(x); +``` + +### 2. Braces Are Required (usually) + +```hemlock +// WRONG +if (x > 0) + print("positive") + +// RIGHT +if (x > 0) { + print("positive"); +} + +// Also OK (single-line statements) +if (x > 0) print("positive"); +``` + +### 3. No Implicit Returns + +```hemlock +// Python: last expression is return value +// def add(a, b): a + b + +// Hemlock: explicit return required +fn add(a: i32, b: i32): i32 { + return a + b; +} + +// Or use expression-bodied function +fn add(a: i32, b: i32): i32 => a + b; +``` + +### 4. Division Always Returns Float + +```hemlock +let x = 7 / 2; // 3.5 (float!) +let y = divi(7, 2); // 3 (integer) +``` + +### 5. String Indexing Returns Bytes + +```hemlock +let s = "hello"; +print(s[0]); // 'h' (rune) +print(s.byte_at(0)); // 104 (ASCII code) + +let emoji = "🚀"; +print(len(emoji)); // 4 (bytes) +print(emoji.length); // 1 (character) +``` + +--- + +## Your First Week Roadmap + +### Day 1-2: Syntax Familiarity +- Write simple programs without memory management +- Use managed arrays and objects +- Get comfortable with Hemlock syntax + +### Day 3-4: Memory Basics +- [Why Manual Memory Matters](./why-memory-matters.md) +- [Exercise 1: First Allocation](./exercises/01-first-allocation.md) +- Practice `alloc`, `free`, `defer` + +### Day 5-6: Deeper Understanding +- [Stack vs Heap](./exercises/02-stack-vs-heap.md) +- [Pointers Explained](./exercises/03-pointers-explained.md) +- Understand ownership and lifetimes + +### Day 7: Practical Application +- Build something small (a command-line tool) +- Mix managed and manual memory as appropriate +- Read the generated C with `hemlockc --keep-c` + +--- + +## Recommended Learning Projects + +### 1. File Counter (Beginner) +Count lines, words, and characters in a file. +```hemlock +// Use file I/O and string operations +// No manual memory needed +``` + +### 2. Simple Allocator (Intermediate) +Build a bump allocator. +```hemlock +// Teaches: memory management internals +// Skills: pointers, arithmetic +``` + +### 3. Thread Pool (Advanced) +Create a pool of workers processing tasks. +```hemlock +// Teaches: concurrency, channels +// Skills: async, spawn, join +``` + +### 4. Mini Database (Advanced) +Store and query records from disk. +```hemlock +// Teaches: file I/O, memory layout +// Skills: serialization, pointers +``` + +--- + +## Getting Help + +- Read error messages carefully - they're designed to help +- Check [Common Mistakes](./common-mistakes.md) for debugging +- Examine generated C code: `hemlockc program.hml --keep-c` +- Print intermediate values liberally + +--- + +*"The best time to learn systems programming was yesterday. The second best time is today."* diff --git a/docs/education/why-memory-matters.md b/docs/education/why-memory-matters.md new file mode 100644 index 00000000..c7086a73 --- /dev/null +++ b/docs/education/why-memory-matters.md @@ -0,0 +1,415 @@ +# Why Manual Memory Matters + +> "You can't truly understand computers until you understand memory." + +This document explains why learning manual memory management is essential for becoming a complete programmer, even if you never write production systems code. + +--- + +## The Hidden Machine + +When you write Python or JavaScript, an enormous amount of work happens invisibly: + +```python +# Python +x = [1, 2, 3, 4, 5] +y = x + [6, 7, 8] +x = None # Is the memory freed? When? How? +``` + +**Questions a systems programmer asks:** +- Where does `[1, 2, 3, 4, 5]` live in memory? +- How many bytes does it occupy? +- When `y = x + [6, 7, 8]`, does it copy or reference `x`? +- When `x = None`, what happens to the original list? +- Why does this simple code potentially allocate memory three times? + +These aren't pedantic questions. Understanding them explains: +- Why your program uses unexpected amounts of RAM +- Why certain operations are slow +- Why memory leaks happen +- How to write efficient code in any language + +--- + +## What Memory Actually Is + +Your computer's memory (RAM) is essentially a giant array of bytes: + +``` +Address: 0x0000 0x0001 0x0002 0x0003 0x0004 ... + ┌──────┬──────┬──────┬──────┬──────┬───── +Value: │ 0x42│ 0x00│ 0x00│ 0x00│ 0xFF│ ... + └──────┴──────┴──────┴──────┴──────┴───── +``` + +Every piece of data in your program lives somewhere in this array. A "pointer" is just the address (index) where the data starts. + +```hemlock +// In Hemlock, you can see this directly +let p = alloc(4); // Get 4 bytes of memory +print(p); // Prints something like: ptr(0x7f8a2b3c4d50) + +ptr_write_i32(p, 42); // Write the number 42 (4 bytes) at that address +let value = ptr_read_i32(p); // Read it back +print(value); // 42 + +free(p); // Return the memory to the system +``` + +--- + +## The Two Regions: Stack and Heap + +Memory is divided into two main regions with very different properties: + +### The Stack: Fast but Limited + +``` +┌─────────────────────────────────┐ +│ THE STACK │ +├─────────────────────────────────┤ +│ • Automatic allocation │ +│ • Very fast (just move pointer)│ +│ • Fixed size (usually ~8MB) │ +│ • LIFO: last in, first out │ +│ • Variables die when function │ +│ returns │ +└─────────────────────────────────┘ +``` + +```hemlock +fn example() { + let x = 42; // x lives on the stack + let y = 100; // y lives on the stack, right after x + // When example() returns, x and y are automatically gone +} +``` + +### The Heap: Flexible but Manual + +``` +┌─────────────────────────────────┐ +│ THE HEAP │ +├─────────────────────────────────┤ +│ • Manual allocation │ +│ • Slower (bookkeeping needed) │ +│ • Limited only by RAM │ +│ • Allocate/free in any order │ +│ • YOU must free it │ +└─────────────────────────────────┘ +``` + +```hemlock +fn example() { + let p = alloc(1000000); // 1MB on the heap + // p (the pointer) is on the stack + // The 1MB of data is on the heap + + // If you don't free(p) before returning, + // you've just leaked 1MB of memory + free(p); +} +``` + +### Why Two Regions? + +**Stack limitations:** +```hemlock +fn cant_return_stack_pointer(): ptr { + let x = 42; + return &x; // WRONG! x dies when function returns + // The pointer becomes invalid ("dangling") +} + +fn correct_heap_return(): ptr { + let p = alloc(4); + ptr_write_i32(p, 42); + return p; // CORRECT: Heap memory survives function return + // Caller must free it eventually +} +``` + +--- + +## What Garbage Collection Hides + +In Python, this "just works": + +```python +def make_list(): + return [1, 2, 3, 4, 5] + +x = make_list() # Where does this memory come from? +x = None # Where does the memory go? +``` + +Behind the scenes, Python's garbage collector: +1. Allocates heap memory for `[1, 2, 3, 4, 5]` +2. Tracks how many references point to it +3. Periodically scans for unreachable objects +4. Frees memory when nothing references it + +**The hidden costs:** +- GC pauses can cause latency spikes +- Reference counting has overhead on every assignment +- Memory isn't freed immediately (unpredictable timing) +- Hard to reason about memory usage + +**In Hemlock, you control everything:** + +```hemlock +fn make_array(): array { + return [1, 2, 3, 4, 5]; // Hemlock arrays are reference-counted +} + +// For manual control, use raw pointers: +fn make_raw_data(): ptr { + let p = alloc(20); // 5 integers × 4 bytes + for (let i = 0; i < 5; i++) { + ptr_write_i32(p + i * 4, i + 1); + } + return p; // Caller must free! +} + +let data = make_raw_data(); +// ... use data ... +free(data); // Explicit, immediate, deterministic +``` + +--- + +## Memory Mistakes: Learning Opportunities + +### 1. Use After Free + +```hemlock +let p = alloc(64); +ptr_write_i32(p, 42); +free(p); + +// MISTAKE: Memory is freed but pointer still exists +let x = ptr_read_i32(p); // Undefined behavior! +``` + +**The lesson:** Freeing memory doesn't invalidate pointers. You must track what's valid. + +**Real-world impact:** Security vulnerabilities. Attackers exploit use-after-free to execute arbitrary code. + +### 2. Double Free + +```hemlock +let p = alloc(64); +free(p); +free(p); // MISTAKE: Already freed! +``` + +**The lesson:** Memory can only be freed once. Freeing twice corrupts the allocator's bookkeeping. + +**Real-world impact:** Crashes, security vulnerabilities, data corruption. + +### 3. Memory Leak + +```hemlock +fn leaky() { + let p = alloc(1000000); // 1MB + // Forgot to free! +} + +// Call this 1000 times = 1GB leaked +for (let i = 0; i < 1000; i++) { + leaky(); +} +``` + +**The lesson:** Every `alloc` needs a matching `free`. No exceptions. + +**Real-world impact:** Programs that slowly consume all system memory. + +### 4. Dangling Pointer + +```hemlock +fn get_pointer(): ptr { + let x = 42; + return &x; // x lives on the stack +} + +let p = get_pointer(); +// x no longer exists! +let value = ptr_read_i32(p); // Reading garbage +``` + +**The lesson:** Stack variables die when their function returns. Never return pointers to them. + +--- + +## Why This Matters (Even If You Stay High-Level) + +### 1. Understanding Performance + +```python +# Python: Why is this slow? +result = "" +for i in range(10000): + result += str(i) # Creates 10,000 new strings! +``` + +With memory knowledge, you understand: each `+=` allocates a new string, copies the old content, adds the new character, and schedules the old string for GC. + +**Efficient version:** +```python +result = "".join(str(i) for i in range(10000)) # One allocation +``` + +### 2. Debugging Memory Issues + +Memory knowledge helps you understand: +- Why your Python program uses 10GB for a 1GB file +- Why Node.js EventEmitters can cause memory leaks +- Why adding items to a list is sometimes O(1) and sometimes O(n) + +### 3. Reading Systems Code + +Linux, databases, web servers, game engines - all written in C/C++. Understanding memory lets you: +- Read and understand this code +- Contribute to these projects +- Debug issues at the boundary between your code and systems code + +### 4. Language Design Appreciation + +Understanding memory helps you appreciate language design tradeoffs: +- Why Rust has the borrow checker +- Why Go's garbage collector is optimized for latency +- Why Java has different GC algorithms to choose from +- Why Python's reference counting + GC combination + +--- + +## Hemlock's Approach: Graduated Safety + +Hemlock gives you options at different safety levels: + +### Level 1: Fully Managed (Safest) + +```hemlock +// Use high-level constructs +let arr = [1, 2, 3]; // Managed array +let str = "hello"; // Managed string +let obj = { x: 1, y: 2 }; // Managed object + +// Memory handled automatically via reference counting +// Similar to Python/JS, but you know it's happening +``` + +### Level 2: Bounds-Checked Buffers + +```hemlock +// Manual allocation with safety rails +let b = buffer(64); +b[0] = 42; // OK +b[100] = 1; // Runtime error: "Buffer index out of bounds" + +// Must still free +free(b); +``` + +### Level 3: Raw Pointers (Full Control) + +```hemlock +// Complete manual control +let p = alloc(64); +ptr_write_i32(p, 42); // No checking +ptr_write_i32(p + 1000, 1); // No checking (will corrupt memory!) +free(p); +``` + +--- + +## Exercises to Build Understanding + +### Exercise 1: Measure Allocation + +```hemlock +import { time_ms } from "@stdlib/time"; + +// How long does allocation take? +let start = time_ms(); +for (let i = 0; i < 100000; i++) { + let p = alloc(100); + free(p); +} +let heap_time = time_ms() - start; +print("100k heap allocs: " + heap_time + "ms"); + +// Compare to "stack allocation" (just variables) +start = time_ms(); +for (let i = 0; i < 100000; i++) { + let x = 42; // Stack - much faster +} +let stack_time = time_ms() - start; +print("100k stack vars: " + stack_time + "ms"); +``` + +### Exercise 2: Find the Leak + +```hemlock +// This program leaks memory. Can you fix it? +fn process_data(data: ptr) { + let result = alloc(100); + // ... do something with result ... + + if (data == ptr_null()) { + return; // BUG: result is never freed on this path + } + + // ... more processing ... + free(result); +} +``` + +### Exercise 3: Implement a Simple Allocator + +```hemlock +// A naive bump allocator - educational, not production! +let heap = buffer(10000); // Our "heap" +let offset = 0; + +fn my_alloc(size: i32): i32 { + let ptr = offset; + offset = offset + size; + if (offset > 10000) { + panic("Out of memory!"); + } + return ptr; // Return offset into our buffer +} + +// Note: No my_free! This allocator can't reclaim memory +// Real allocators are much more complex +``` + +--- + +## Key Takeaways + +1. **Memory is finite and must be managed** - Either you manage it or a garbage collector does, but someone has to. + +2. **Pointers are just addresses** - Demystify them: they're numbers that tell you where data lives. + +3. **Stack is automatic, heap is manual** - Know which you're using and why. + +4. **Every allocation needs deallocation** - Memory doesn't free itself. + +5. **Mistakes are learning opportunities** - Crashes teach you how systems really work. + +--- + +## Next Steps + +Ready to apply this knowledge? Continue to: +- [Your First Allocation](./exercises/01-first-allocation.md) - Hands-on memory exercises +- [Common Memory Mistakes](./common-mistakes.md) - Learn to debug memory issues +- [Stack vs Heap Deep Dive](./exercises/02-stack-vs-heap.md) - More on memory regions + +--- + +*"The computer is doing exactly what you told it to do. Learning memory teaches you to be precise about what you ask for."* diff --git a/examples/educational/README.md b/examples/educational/README.md new file mode 100644 index 00000000..4d6bfb50 --- /dev/null +++ b/examples/educational/README.md @@ -0,0 +1,37 @@ +# Educational Examples + +These examples are designed to teach systems programming concepts. Each includes detailed comments explaining what's happening and why. + +## Examples + +### memory_journey.hml +**Concepts:** Memory allocation, pointers, pointer arithmetic, memory dangers, building data structures + +A step-by-step journey through memory management, culminating in building a dynamic array from scratch. + +```bash +./hemlock examples/educational/memory_journey.hml +``` + +### concurrency_basics.hml +**Concepts:** Tasks, parallelism, race conditions, atomics, channels + +Learn concurrent programming through practical examples, including demonstrations of race conditions and how to fix them. + +```bash +./hemlock examples/educational/concurrency_basics.hml +``` + +## Learning Path + +1. Start with `memory_journey.hml` to understand memory fundamentals +2. Move to `concurrency_basics.hml` to learn parallel programming +3. Review the exercises in `docs/education/exercises/` +4. Follow the full curriculum in `docs/education/curriculum.md` + +## Related Documentation + +- [Education Overview](../../docs/education/README.md) +- [Why Manual Memory Matters](../../docs/education/why-memory-matters.md) +- [Common Mistakes](../../docs/education/common-mistakes.md) +- [Learning Curriculum](../../docs/education/curriculum.md) diff --git a/examples/educational/concurrency_basics.hml b/examples/educational/concurrency_basics.hml new file mode 100644 index 00000000..c377d972 --- /dev/null +++ b/examples/educational/concurrency_basics.hml @@ -0,0 +1,360 @@ +// ============================================================================ +// HEMLOCK EDUCATIONAL EXAMPLE: Concurrency Basics +// ============================================================================ +// +// This program teaches concurrent programming concepts through practical +// examples. Each section builds on the previous one. +// +// Run with: ./hemlock examples/educational/concurrency_basics.hml +// ============================================================================ + +import { sleep, time_ms } from "@stdlib/time"; + +print("=== CONCURRENCY BASICS ==="); +print(""); + +// ============================================================================ +// LESSON 1: What is Concurrency? +// ============================================================================ +// +// Concurrency means multiple things happening at once (or appearing to). +// In Hemlock, `spawn` creates a real OS thread that runs in parallel. + +print("--- LESSON 1: Your First Task ---"); +print(""); + +// A regular function runs on the main thread +fn regular_work() { + print(" Regular: Starting work"); + sleep(100); // Sleep 100ms + print(" Regular: Done"); +} + +// An async function CAN be spawned to run on another thread +async fn async_work() { + print(" Async: Starting work"); + sleep(100); + print(" Async: Done"); +} + +print("Running regular function (blocks main thread):"); +let start = time_ms(); +regular_work(); +print(" Time: " + (time_ms() - start) + "ms"); +print(""); + +print("Running async function with spawn (runs in parallel):"); +start = time_ms(); +let task = spawn(async_work); +print(" Main thread continues immediately!"); +print(" Main thread time: " + (time_ms() - start) + "ms"); +join(task); // Wait for the async task to finish +print(" Total time: " + (time_ms() - start) + "ms"); +print(""); + +// ============================================================================ +// LESSON 2: Parallelism Speedup +// ============================================================================ +// +// The power of concurrency: do multiple things simultaneously. + +print("--- LESSON 2: Parallelism Speedup ---"); +print(""); + +async fn slow_computation(id: i32): i32 { + print(" Task " + id + ": starting computation"); + sleep(200); // Simulates 200ms of work + print(" Task " + id + ": finished"); + return id * 100; +} + +// Sequential: Run one after another (takes ~800ms) +print("Sequential execution (4 tasks, 200ms each):"); +start = time_ms(); +let results_seq = []; +for (let i = 1; i <= 4; i++) { + results_seq.push(join(spawn(slow_computation, i))); +} +print(" Results: " + results_seq.join(", ")); +print(" Time: " + (time_ms() - start) + "ms (expected ~800ms)"); +print(""); + +// Parallel: Run all at once (takes ~200ms) +print("Parallel execution (4 tasks at once):"); +start = time_ms(); + +// Spawn all tasks first +let tasks = []; +for (let i = 1; i <= 4; i++) { + tasks.push(spawn(slow_computation, i)); +} + +// Then wait for all of them +let results_par = []; +for (t in tasks) { + results_par.push(join(t)); +} +print(" Results: " + results_par.join(", ")); +print(" Time: " + (time_ms() - start) + "ms (expected ~200ms)"); +print(""); + +// ============================================================================ +// LESSON 3: The Race Condition Problem +// ============================================================================ +// +// When multiple threads access shared data without synchronization, +// they can interfere with each other. This is called a "race condition". + +print("--- LESSON 3: Race Conditions ---"); +print(""); + +// DANGER: This code has a race condition! +// Multiple threads read-modify-write a shared variable. +print("Demonstrating a race condition:"); +print(" (This may give wrong results!)"); +print(""); + +// We'll use a pointer to shared memory +let shared = alloc(sizeof(i32)); +ptr_write_i32(shared, 0); + +async fn increment_shared(p: ptr, iterations: i32) { + for (let i = 0; i < iterations; i++) { + // RACE: read, add, write is NOT atomic + let current = ptr_read_i32(p); + ptr_write_i32(p, current + 1); + } +} + +// Run two tasks, each incrementing 1000 times +// Expected result: 2000 (if no race) +let t1 = spawn(increment_shared, shared, 1000); +let t2 = spawn(increment_shared, shared, 1000); +join(t1); +join(t2); + +let result = ptr_read_i32(shared); +print(" Two tasks, each incrementing 1000 times"); +print(" Expected result: 2000"); +print(" Actual result: " + result); +if (result != 2000) { + print(" RACE DETECTED! Some increments were lost."); + print(" This happens because:"); + print(" Thread A reads: 500"); + print(" Thread B reads: 500"); + print(" Thread A writes: 501"); + print(" Thread B writes: 501 <- Thread A's increment is lost!"); +} +print(""); + +free(shared); + +// ============================================================================ +// LESSON 4: Fixing Races with Atomics +// ============================================================================ +// +// Atomic operations are indivisible - they complete entirely or not at all. +// No other thread can see a half-done atomic operation. + +print("--- LESSON 4: Atomic Operations ---"); +print(""); + +let atomic_counter = alloc(sizeof(i32)); +atomic_store_i32(atomic_counter, 0); + +async fn increment_atomic(p: ptr, iterations: i32) { + for (let i = 0; i < iterations; i++) { + // atomic_add_i32 is ATOMIC: read-modify-write happens together + atomic_add_i32(p, 1); + } +} + +// Same test, but with atomic operations +let t3 = spawn(increment_atomic, atomic_counter, 1000); +let t4 = spawn(increment_atomic, atomic_counter, 1000); +join(t3); +join(t4); + +let atomic_result = atomic_load_i32(atomic_counter); +print(" Using atomic_add_i32:"); +print(" Expected result: 2000"); +print(" Actual result: " + atomic_result); +if (atomic_result == 2000) { + print(" SUCCESS! Atomics prevented the race."); +} +print(""); + +free(atomic_counter); + +// ============================================================================ +// LESSON 5: Channels for Communication +// ============================================================================ +// +// Instead of sharing memory, we can send messages through channels. +// This is often safer and easier to reason about. + +print("--- LESSON 5: Channels ---"); +print(""); + +print("Channels let tasks communicate by sending messages."); +print(""); + +// Create a channel with buffer size 5 +let ch = channel(5); + +// Producer: sends data into the channel +async fn producer(ch: channel, count: i32) { + print(" Producer: starting"); + for (let i = 1; i <= count; i++) { + ch.send(i * 10); + print(" Producer: sent " + (i * 10)); + } + ch.close(); // Signal we're done + print(" Producer: closed channel"); +} + +// Consumer: receives data from the channel +async fn consumer(ch: channel): i32 { + print(" Consumer: starting"); + let sum = 0; + loop { + let value = ch.recv(); + if (value == null) { + // Channel closed + break; + } + print(" Consumer: received " + value); + sum = sum + value; + } + print(" Consumer: channel closed, sum = " + sum); + return sum; +} + +let prod = spawn(producer, ch, 5); +let cons = spawn(consumer, ch); + +join(prod); +let total = join(cons); + +print(" Final sum: " + total); +print(""); + +// ============================================================================ +// LESSON 6: Memory Ownership with Tasks +// ============================================================================ +// +// CRITICAL: When you pass a pointer to a spawned task, you must ensure +// the memory remains valid until the task completes! + +print("--- LESSON 6: Memory and Tasks ---"); +print(""); + +print("RULE: Never free memory while a task is using it!"); +print(""); + +// CORRECT: Wait for task before freeing +async fn use_memory(p: ptr): i32 { + sleep(50); // Simulate some work + return ptr_read_i32(p); +} + +let safe_ptr = alloc(sizeof(i32)); +ptr_write_i32(safe_ptr, 12345); + +let mem_task = spawn(use_memory, safe_ptr); +// DO NOT free here - task is still running! + +let mem_result = join(mem_task); // Wait for task +print("Task returned: " + mem_result); + +free(safe_ptr); // NOW it's safe to free +print("Safely freed memory after task completed"); +print(""); + +// ============================================================================ +// LESSON 7: Practical Example - Parallel Sum +// ============================================================================ +// +// Let's put it all together: sum a large array using multiple threads. + +print("--- LESSON 7: Parallel Sum Example ---"); +print(""); + +// Create an array to sum +let arr_size = 100; +let arr_bytes = arr_size * sizeof(i32); +let array = alloc(arr_bytes); + +// Fill with values 1..100 +for (let i = 0; i < arr_size; i++) { + ptr_write_i32(array + i * sizeof(i32), i + 1); +} +print("Created array with values 1 to " + arr_size); + +// Sequential sum for comparison +let seq_sum = 0; +start = time_ms(); +for (let i = 0; i < arr_size; i++) { + seq_sum = seq_sum + ptr_read_i32(array + i * sizeof(i32)); +} +let seq_time = time_ms() - start; +print("Sequential sum: " + seq_sum + " (" + seq_time + "ms)"); + +// Parallel sum: divide work among threads +async fn partial_sum(arr: ptr, start_idx: i32, end_idx: i32): i32 { + let sum = 0; + for (let i = start_idx; i < end_idx; i++) { + sum = sum + ptr_read_i32(arr + i * sizeof(i32)); + } + return sum; +} + +start = time_ms(); +let num_threads = 4; +let chunk_size = arr_size / num_threads; + +let sum_tasks = []; +for (let t = 0; t < num_threads; t++) { + let start_idx = t * chunk_size; + let end_idx = start_idx + chunk_size; + if (t == num_threads - 1) { + end_idx = arr_size; // Last thread gets remainder + } + sum_tasks.push(spawn(partial_sum, array, start_idx, end_idx)); +} + +// Collect results +let par_sum = 0; +for (task in sum_tasks) { + par_sum = par_sum + join(task); +} +let par_time = time_ms() - start; +print("Parallel sum (" + num_threads + " threads): " + par_sum + " (" + par_time + "ms)"); + +free(array); +print(""); + +// ============================================================================ +// SUMMARY +// ============================================================================ + +print("--- SUMMARY ---"); +print(""); +print("What you learned:"); +print(" 1. spawn() creates real parallel threads"); +print(" 2. join() waits for a task to complete"); +print(" 3. Race conditions occur when threads share unsynchronized data"); +print(" 4. Atomic operations (atomic_add_i32, etc.) prevent races"); +print(" 5. Channels provide safe inter-task communication"); +print(" 6. Never free memory while tasks are using it"); +print(""); +print("Key functions:"); +print(" spawn(fn, args...) - Start a task"); +print(" join(task) - Wait and get result"); +print(" detach(task) - Let task run independently"); +print(" channel(size) - Create a channel"); +print(" ch.send(value) - Send through channel"); +print(" ch.recv() - Receive from channel"); +print(" atomic_* - Lock-free atomic operations"); +print(""); +print("=== END OF CONCURRENCY BASICS ==="); diff --git a/examples/educational/memory_journey.hml b/examples/educational/memory_journey.hml new file mode 100644 index 00000000..e6dfcad1 --- /dev/null +++ b/examples/educational/memory_journey.hml @@ -0,0 +1,326 @@ +// ============================================================================ +// HEMLOCK EDUCATIONAL EXAMPLE: A Journey Through Memory +// ============================================================================ +// +// This program teaches fundamental memory management concepts by building +// a simple dynamic array from scratch. Each section introduces a new concept. +// +// Run with: ./hemlock examples/educational/memory_journey.hml +// ============================================================================ + +print("=== HEMLOCK MEMORY JOURNEY ==="); +print(""); + +// ============================================================================ +// LESSON 1: What is Memory? +// ============================================================================ +// +// Your computer's memory (RAM) is like a giant array of numbered mailboxes. +// Each mailbox holds one byte (8 bits, values 0-255). +// The mailbox number is called an "address". +// +// When you allocate memory, you're requesting some contiguous mailboxes. + +print("--- LESSON 1: Basic Allocation ---"); + +// alloc(n) requests n bytes of memory and returns the address of the first byte +let p = alloc(4); // Request 4 bytes (enough for one 32-bit integer) + +print("Allocated 4 bytes at address: " + p); +print(" This is our 'mailbox number' - where our data will live"); + +// We must ALWAYS free what we allocate +free(p); +print("Freed the memory - returned our mailboxes to the system"); +print(""); + +// ============================================================================ +// LESSON 2: Writing and Reading Data +// ============================================================================ +// +// Memory is just bytes. To store meaningful data, we need to: +// 1. Allocate enough bytes for our data type +// 2. Write the data (encoding it as bytes) +// 3. Read it back (decoding the bytes) + +print("--- LESSON 2: Writing and Reading ---"); + +// An i32 (32-bit integer) needs 4 bytes +let size_of_int = sizeof(i32); +print("Size of i32: " + size_of_int + " bytes"); + +let ptr = alloc(size_of_int); +print("Allocated " + size_of_int + " bytes for one integer"); + +// Write an integer value +ptr_write_i32(ptr, 42); +print("Wrote value 42 to memory"); + +// Read it back +let value = ptr_read_i32(ptr); +print("Read back: " + value); + +// What's actually in those 4 bytes? +print("The 4 bytes (little-endian):"); +for (let i = 0; i < 4; i++) { + let byte = ptr_read_u8(ptr + i); + print(" Byte " + i + ": " + byte); +} +// 42 in binary is 00101010, which is just byte 0 = 42, bytes 1-3 = 0 + +free(ptr); +print(""); + +// ============================================================================ +// LESSON 3: Pointer Arithmetic +// ============================================================================ +// +// A pointer is just a number (the address). Adding to it moves to a later +// address. This lets us store multiple values sequentially. + +print("--- LESSON 3: Pointer Arithmetic (Building an Array) ---"); + +let count = 5; +let elem_size = sizeof(i32); +let total_bytes = count * elem_size; + +print("Creating space for " + count + " integers:"); +print(" Each integer: " + elem_size + " bytes"); +print(" Total needed: " + total_bytes + " bytes"); + +let arr_ptr = alloc(total_bytes); +print("Allocated at: " + arr_ptr); + +// Write values: 10, 20, 30, 40, 50 +print("Writing values..."); +for (let i = 0; i < count; i++) { + let offset = i * elem_size; // How many bytes from start + let addr = arr_ptr + offset; // The actual address + let value = (i + 1) * 10; // Values: 10, 20, 30, 40, 50 + + ptr_write_i32(addr, value); + print(" arr[" + i + "] at offset " + offset + " = " + value); +} + +// Read them back +print("Reading values..."); +let sum = 0; +for (let i = 0; i < count; i++) { + let value = ptr_read_i32(arr_ptr + i * elem_size); + sum = sum + value; + print(" arr[" + i + "] = " + value); +} +print("Sum: " + sum); + +free(arr_ptr); +print(""); + +// ============================================================================ +// LESSON 4: The Danger of Manual Memory +// ============================================================================ +// +// Manual memory management is powerful but dangerous. Here are patterns +// that cause bugs. (We'll demonstrate safe versions.) + +print("--- LESSON 4: Memory Dangers (and How to Avoid Them) ---"); + +// DANGER 1: Use-after-free +// Once you free memory, it may be reused by the next allocation! +print(""); +print("Danger 1: Use-after-free"); +let danger1 = alloc(4); +ptr_write_i32(danger1, 999); +print(" Before free: " + ptr_read_i32(danger1)); +free(danger1); +// DON'T DO: print(ptr_read_i32(danger1)); // Would read freed memory! +print(" After free: memory is invalid, we must not read it"); +print(" SOLUTION: Set pointer to null after freeing"); +danger1 = ptr_null(); +print(" Pointer is now: " + danger1); + +// DANGER 2: Memory leak +// If you lose track of a pointer, the memory can never be freed +print(""); +print("Danger 2: Memory leaks"); +fn leaky_function() { + let leak = alloc(100); + // Forgot to free! + // When this function returns, we lose the pointer + // The 100 bytes are leaked forever +} +print(" A leaky function allocates but doesn't free"); +print(" SOLUTION: Use 'defer' for guaranteed cleanup"); + +fn safe_function() { + let safe = alloc(100); + defer free(safe); // Will run when function exits, no matter how + + // Even if we return early or throw an error, defer runs + print(" Inside safe_function: allocated and deferred free"); +} +safe_function(); +print(" After safe_function: memory was freed by defer"); + +// DANGER 3: Double-free +// Freeing the same memory twice corrupts the allocator +print(""); +print("Danger 3: Double-free"); +let danger3 = alloc(4); +free(danger3); +// DON'T DO: free(danger3); // Would crash or corrupt memory! +print(" After first free, we must not free again"); +print(" SOLUTION: Set to null, check before free"); +danger3 = ptr_null(); +if (danger3 != ptr_null()) { + free(danger3); +} +print(" With null check, accidental double-free is prevented"); +print(""); + +// ============================================================================ +// LESSON 5: Building a Dynamic Array +// ============================================================================ +// +// Now let's apply what we learned to build a growable array! + +print("--- LESSON 5: Building a Dynamic Array ---"); + +// Our "object" is just organized memory: +// - bytes 0-3: capacity (how many elements we can hold) +// - bytes 4-7: length (how many elements we have) +// - bytes 8+: the actual data + +fn dynarray_create(initial_capacity: i32): ptr { + // Allocate: 8 bytes for header + capacity * 4 bytes for data + let header_size = 8; + let data_size = initial_capacity * sizeof(i32); + let total = header_size + data_size; + + let arr = alloc(total); + + // Initialize header + ptr_write_i32(arr, initial_capacity); // capacity at offset 0 + ptr_write_i32(arr + 4, 0); // length at offset 4 + + print("Created dynamic array:"); + print(" Capacity: " + initial_capacity); + print(" Length: 0"); + print(" Total bytes: " + total); + + return arr; +} + +fn dynarray_capacity(arr: ptr): i32 { + return ptr_read_i32(arr); +} + +fn dynarray_length(arr: ptr): i32 { + return ptr_read_i32(arr + 4); +} + +fn dynarray_get(arr: ptr, index: i32): i32 { + let len = dynarray_length(arr); + if (index < 0 || index >= len) { + panic("Index out of bounds: " + index + " (length: " + len + ")"); + } + // Data starts at offset 8 + return ptr_read_i32(arr + 8 + index * sizeof(i32)); +} + +fn dynarray_push(ref arr: ptr, value: i32) { + let cap = dynarray_capacity(arr); + let len = dynarray_length(arr); + + // Need to grow? + if (len >= cap) { + let new_cap = cap * 2; + let new_size = 8 + new_cap * sizeof(i32); + + print(" Growing array: " + cap + " -> " + new_cap); + + let new_arr = alloc(new_size); + + // Copy header + ptr_write_i32(new_arr, new_cap); + ptr_write_i32(new_arr + 4, len); + + // Copy data + memcpy(new_arr + 8, arr + 8, len * sizeof(i32)); + + // Free old, use new + free(arr); + arr = new_arr; + } + + // Write the new value + ptr_write_i32(arr + 8 + len * sizeof(i32), value); + + // Increment length + ptr_write_i32(arr + 4, len + 1); +} + +fn dynarray_print(arr: ptr) { + let len = dynarray_length(arr); + let cap = dynarray_capacity(arr); + + print("DynArray (length=" + len + ", capacity=" + cap + "):"); + let str = " ["; + for (let i = 0; i < len; i++) { + if (i > 0) { + str = str + ", "; + } + str = str + dynarray_get(arr, i); + } + str = str + "]"; + print(str); +} + +fn dynarray_free(arr: ptr) { + free(arr); + print("Freed dynamic array"); +} + +// Let's use our dynamic array! +print(""); +print("Using our dynamic array:"); + +let my_arr = dynarray_create(4); // Start with capacity 4 + +// Push some values (this will trigger growth) +for (let i = 1; i <= 10; i++) { + print("Pushing " + i + "..."); + dynarray_push(my_arr, i * 100); +} + +dynarray_print(my_arr); + +// Access specific elements +print(""); +print("Accessing elements:"); +print(" my_arr[0] = " + dynarray_get(my_arr, 0)); +print(" my_arr[4] = " + dynarray_get(my_arr, 4)); +print(" my_arr[9] = " + dynarray_get(my_arr, 9)); + +dynarray_free(my_arr); +print(""); + +// ============================================================================ +// LESSON 6: What You've Learned +// ============================================================================ + +print("--- LESSON 6: Summary ---"); +print(""); +print("You've learned:"); +print(" 1. Memory is bytes at numbered addresses"); +print(" 2. alloc() gets memory, free() returns it"); +print(" 3. Pointer arithmetic lets you access sequential data"); +print(" 4. Common bugs: use-after-free, leaks, double-free"); +print(" 5. 'defer' provides automatic cleanup"); +print(" 6. You can build data structures from raw memory!"); +print(""); +print("Next steps:"); +print(" - Try the exercises in docs/education/exercises/"); +print(" - Read docs/education/common-mistakes.md"); +print(" - Build something yourself!"); +print(""); +print("=== END OF MEMORY JOURNEY ===");