[PW_SID:955410] LKMM generic atomics in Rust#323
Conversation
In order to support LKMM atomics in Rust, add rust_helper_* for atomic APIs. These helpers ensure the implementation of LKMM atomics in Rust is the same as in C. This could save the maintenance burden of having two similar atomic implementations in asm. Originally-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Boqun Feng <boqun.feng@gmail.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Preparation for generic atomic implementation. To unify the implementation of a generic method over `i32` and `i64`, the C side atomic methods need to be grouped so that in a generic method, they can be referred as <type>::<method>, otherwise their parameters and return value are different between `i32` and `i64`, which would require using `transmute()` to unify the type into a `T`. Introduce `AtomicImpl` to represent a basic type in Rust that has the direct mapping to an atomic implementation from C. This trait is sealed, and currently only `i32` and `i64` impl this. Further, different methods are put into different `*Ops` trait groups, and this is for the future when smaller types like `i8`/`i16` are supported but only with a limited set of API (e.g. only set(), load(), xchg() and cmpxchg(), no add() or sub() etc). While the atomic mod is introduced, documentation is also added for memory models and data races. Also bump my role to the maintainer of ATOMIC INFRASTRUCTURE to reflect my responsiblity on the Rust atomic mod. Signed-off-by: Boqun Feng <boqun.feng@gmail.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Preparation for atomic primitives. Instead of a suffix like _acquire, a
method parameter along with the corresponding generic parameter will be
used to specify the ordering of an atomic operations. For example,
atomic load() can be defined as:
impl<T: ...> Atomic<T> {
pub fn load<O: AcquireOrRelaxed>(&self, _o: O) -> T { ... }
}
and acquire users would do:
let r = x.load(Acquire);
relaxed users:
let r = x.load(Relaxed);
doing the following:
let r = x.load(Release);
will cause a compiler error.
Compared to suffixes, it's easier to tell what ordering variants an
operation has, and it also make it easier to unify the implementation of
all ordering variants in one method via generic. The `IS_RELAXED` and
`ORDER` associate consts are for generic function to pick up the
particular implementation specified by an ordering annotation.
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
To provide using LKMM atomics for Rust code, a generic `Atomic<T>` is added, currently `T` needs to be Send + Copy because these are the straightforward usages and all basic types support this. The trait `AllowAtomic` should be only implemented inside atomic mod until the generic atomic framework is mature enough (unless the implementer is a `#[repr(transparent)]` new type). `AtomicImpl` types are automatically `AllowAtomic`, and so far only basic operations load() and store() are introduced. Signed-off-by: Boqun Feng <boqun.feng@gmail.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
xchg() and cmpxchg() are basic operations on atomic. Provide these based on C APIs. Note that cmpxchg() use the similar function signature as compare_exchange() in Rust std: returning a `Result`, `Ok(old)` means the operation succeeds and `Err(old)` means the operation fails. With the compiler optimization and inline helpers, it should provides the same efficient code generation as using atomic_try_cmpxchg() or atomic_cmpxchg() correctly. Except it's not! Because of commit 44fe844 ("locking/atomic: Fix atomic_try_cmpxchg() semantics"), the atomic_try_cmpxchg() on x86 has a branch even if the caller doesn't care about the success of cmpxchg and only wants to use the old value. For example, for code like: // Uses the latest value regardlessly, same as atomic_cmpxchg() in C. let latest = x.cmpxchg(42, 64, Full).unwrap_or_else(|old| old); It will still generate code: movl $0x40, %ecx movl $0x34, %eax lock cmpxchgl %ecx, 0x4(%rsp) jne 1f 2: ... 1: movl %eax, %ecx jmp 2b Attempting to write an x86 try_cmpxchg_exclusive() for Rust use only, because the Rust function takes a `&mut` for old pointer, which must be exclusive to the function, therefore it's unsafe to use some shared pointer. But maybe I'm missing something? Signed-off-by: Boqun Feng <boqun.feng@gmail.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
One important set of atomic operations is the arithmetic operations, i.e. add(), sub(), fetch_add(), add_return(), etc. However it may not make senses for all the types that `AllowAtomic` to have arithmetic operations, for example a `Foo(u32)` may not have a reasonable add() or sub(), plus subword types (`u8` and `u16`) currently don't have atomic arithmetic operations even on C side and might not have them in the future in Rust (because they are usually suboptimal on a few architecures). Therefore add a subtrait of `AllowAtomic` describing which types have and can do atomic arithemtic operations. A few things about this `AllowAtomicArithmetic` trait: * It has an associate type `Delta` instead of using `AllowAllowAtomic::Repr` because, a `Bar(u32)` (whose `Repr` is `i32`) may not wants an `add(&self, i32)`, but an `add(&self, u32)`. * `AtomicImpl` types already implement an `AtomicHasArithmeticOps` trait, so add blanket implementation for them. In the future, `i8` and `i16` may impl `AtomicImpl` but not `AtomicHasArithmeticOps` if arithemtic operations are not available. Only add() and fetch_add() are added. The rest will be added in the future. Signed-off-by: Boqun Feng <boqun.feng@gmail.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Add generic atomic support for basic unsigned types that have an `AtomicImpl` with the same size and alignment. Signed-off-by: Boqun Feng <boqun.feng@gmail.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Add generic atomic support for `usize` and `isize`. Note that instead of mapping directly to `atomic_long_t`, the represention type (`AllowAtomic::Repr`) is selected based on CONFIG_64BIT. This reduces the necessarity of creating `atomic_long_*` helpers, which could save the binary size of kernel if inline helpers are not available. Signed-off-by: Boqun Feng <boqun.feng@gmail.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Add atomic support for raw pointer values, similar to `isize` and `usize`, the representation type is selected based on CONFIG_64BIT. `*mut T` is not `Send`, however `Atomic<*mut T>` definitely needs to be a `Sync`, and that's the whole point of atomics: being able to have multiple shared references in different threads so that they can sync with each other. As a result, a pointer value will be transferred from one thread to another via `Atomic<*mut T>`: <thread 1> <thread 2> x.store(p1, Relaxed); let p = x.load(p1, Relaxed); This means a raw pointer value (`*mut T`) needs to be able to transfer across thread boundaries, which is essentially `Send`. To reflect this in the type system, and based on the fact that pointer values can be transferred safely (only using them to dereference is unsafe), as suggested by Alice, extend the `AllowAtomic` trait to include a customized `Send` semantics, that is: `impl AllowAtomic` has to be safe to be transferred across thread boundaries. Suggested-by: Alice Ryhl <aliceryhl@google.com> Signed-off-by: Boqun Feng <boqun.feng@gmail.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
(This is more an RFC) Add arithmetic operations support for `Atomic<*mut T>`. Currently the semantics of arithmetic atomic operation is the same as pointer arithmetic, that is, e.g. `Atomic<*mut u64>::add(1)` is adding 8 (`size_of::<u64>`) to the pointer value. In Rust std library, there are two sets of pointer arithmetic for `AtomicPtr`: * ptr_add() and ptr_sub(), which is the same as Atomic<*mut T>::add(), pointer arithmetic. * byte_add() and byte_sub(), which use the input as byte offset to change the pointer value, e.g. byte_add(1) means adding 1 to the pointer value. We can either take the approach in the current patch and add byte_add() later on if needed, or start with ptr_add() and byte_add() naming. Signed-off-by: Boqun Feng <boqun.feng@gmail.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Memory barriers are building blocks for concurrent code, hence provide a minimal set of them. The compiler barrier, barrier(), is implemented in inline asm instead of using core::sync::atomic::compiler_fence() because memory models are different: kernel's atomics are implemented in inline asm therefore the compiler barrier should be implemented in inline asm as well. Signed-off-by: Boqun Feng <boqun.feng@gmail.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
RCU protected pointers are an atomic pointer that can be loaded and dereferenced by mulitple RCU readers, but only one updater/writer can change the value (following a read-copy-update pattern usually). This is useful in the case where data is read-mostly. The rationale of this patch is to provide a proof of concept on how RCU should be exposed to the Rust world, and it also serves as an example for atomic usage. Similar mechanisms like ArcSwap [1] are already widely used. Provide a `Rcu<P>` type with an atomic pointer implementation. `P` has to be a `ForeignOwnable`, which means the ownership of a object can be represented by a pointer-size value. `Rcu::dereference()` requires a RCU Guard, which means dereferencing is only valid under RCU read lock protection. `Rcu::read_copy_update()` is the operation for updaters, it requries a `Pin<&mut Self>` for exclusive accesses, since RCU updaters are normally exclusive with each other. A lot of RCU functionalities including asynchronously free (call_rcu() and kfree_rcu()) are still missing, and will be the future work. Also, we still need language changes like field projection [2] to provide better ergonomic. Acknowledgment: this work is based on a lot of productive discussions and hard work from others, these are the ones I can remember (sorry if I forgot your contribution): * Wedson started the work on RCU field projection and Benno followed it up and had been working on it as a more general language feature. Also, Gary's field-projection repo [3] has been used as an example for related discussions. * During Kangrejos 2023 [4], Gary, Benno and Alice provided a lot of feedbacks on the talk from Paul and me: "If you want to use RCU in Rust for Linux kernel..." * During a recent discussion among Benno, Paul and me, Benno suggested using `Pin<&mut>` to guarantee the exclusive access on updater operations. Link: https://crates.io/crates/arc-swap [1] Link: https://rust-lang.zulipchat.com/#narrow/channel/213817-t-lang/topic/Field.20Projections/near/474648059 [2] Link: https://github.com/nbdd0121/field-projection [3] Link: https://kangrejos.com/2023 [4] Signed-off-by: Boqun Feng <boqun.feng@gmail.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
|
Patch 1: "[RFC,v3,01/12] rust: Introduce atomic API helpers" |
|
Patch 1: "[RFC,v3,01/12] rust: Introduce atomic API helpers" |
|
Patch 1: "[RFC,v3,01/12] rust: Introduce atomic API helpers" |
|
Patch 1: "[RFC,v3,01/12] rust: Introduce atomic API helpers" |
|
Patch 1: "[RFC,v3,01/12] rust: Introduce atomic API helpers" |
|
Patch 1: "[RFC,v3,01/12] rust: Introduce atomic API helpers" |
|
Patch 1: "[RFC,v3,01/12] rust: Introduce atomic API helpers" |
|
Patch 1: "[RFC,v3,01/12] rust: Introduce atomic API helpers" |
|
Patch 1: "[RFC,v3,01/12] rust: Introduce atomic API helpers" |
|
Patch 1: "[RFC,v3,01/12] rust: Introduce atomic API helpers" |
|
Patch 1: "[RFC,v3,01/12] rust: Introduce atomic API helpers" |
|
Patch 1: "[RFC,v3,01/12] rust: Introduce atomic API helpers" |
|
Patch 2: "[RFC,v3,02/12] rust: sync: Add basic atomic operation mapping framework" |
|
Patch 2: "[RFC,v3,02/12] rust: sync: Add basic atomic operation mapping framework" |
|
Patch 2: "[RFC,v3,02/12] rust: sync: Add basic atomic operation mapping framework" |
|
Patch 2: "[RFC,v3,02/12] rust: sync: Add basic atomic operation mapping framework" |
|
Patch 2: "[RFC,v3,02/12] rust: sync: Add basic atomic operation mapping framework" |
|
Patch 2: "[RFC,v3,02/12] rust: sync: Add basic atomic operation mapping framework" |
|
Patch 10: "[RFC,v3,10/12] rust: sync: atomic: Add arithmetic ops for Atomic<*mut T>" |
|
Patch 10: "[RFC,v3,10/12] rust: sync: atomic: Add arithmetic ops for Atomic<*mut T>" |
|
Patch 10: "[RFC,v3,10/12] rust: sync: atomic: Add arithmetic ops for Atomic<*mut T>" |
|
Patch 10: "[RFC,v3,10/12] rust: sync: atomic: Add arithmetic ops for Atomic<*mut T>" |
|
Patch 11: "[RFC,v3,11/12] rust: sync: Add memory barriers" |
|
Patch 11: "[RFC,v3,11/12] rust: sync: Add memory barriers" |
|
Patch 11: "[RFC,v3,11/12] rust: sync: Add memory barriers" |
|
Patch 11: "[RFC,v3,11/12] rust: sync: Add memory barriers" |
|
Patch 11: "[RFC,v3,11/12] rust: sync: Add memory barriers" |
|
Patch 11: "[RFC,v3,11/12] rust: sync: Add memory barriers" |
|
Patch 11: "[RFC,v3,11/12] rust: sync: Add memory barriers" |
|
Patch 11: "[RFC,v3,11/12] rust: sync: Add memory barriers" |
|
Patch 11: "[RFC,v3,11/12] rust: sync: Add memory barriers" |
|
Patch 11: "[RFC,v3,11/12] rust: sync: Add memory barriers" |
|
Patch 11: "[RFC,v3,11/12] rust: sync: Add memory barriers" |
|
Patch 11: "[RFC,v3,11/12] rust: sync: Add memory barriers" |
|
Patch 12: "[RFC,v3,12/12] rust: sync: rcu: Add RCU protected pointer" |
|
Patch 12: "[RFC,v3,12/12] rust: sync: rcu: Add RCU protected pointer" |
|
Patch 12: "[RFC,v3,12/12] rust: sync: rcu: Add RCU protected pointer" |
|
Patch 12: "[RFC,v3,12/12] rust: sync: rcu: Add RCU protected pointer" |
|
Patch 12: "[RFC,v3,12/12] rust: sync: rcu: Add RCU protected pointer" |
|
Patch 12: "[RFC,v3,12/12] rust: sync: rcu: Add RCU protected pointer" |
|
Patch 12: "[RFC,v3,12/12] rust: sync: rcu: Add RCU protected pointer" |
|
Patch 12: "[RFC,v3,12/12] rust: sync: rcu: Add RCU protected pointer" |
|
Patch 12: "[RFC,v3,12/12] rust: sync: rcu: Add RCU protected pointer" |
|
Patch 12: "[RFC,v3,12/12] rust: sync: rcu: Add RCU protected pointer" |
|
Patch 12: "[RFC,v3,12/12] rust: sync: rcu: Add RCU protected pointer" |
|
Patch 12: "[RFC,v3,12/12] rust: sync: rcu: Add RCU protected pointer" |
PR for series 955410 applied to workflow__riscv__fixes
Name: LKMM generic atomics in Rust
URL: https://patchwork.kernel.org/project/linux-riscv/list/?series=955410
Version: 3