Skip to content

ColloidOutput

Jump to bottom
Sebastian Schmieschek edited this page Aug 12, 2015 · 2 revisions

= Output of colloid information == Objectives

  • The format of the output file must be portable and, preferably, binary.

  • The layout of the output file must be independent of the number of ranks (optional).

  • Enough information should be output to be used as a checkpoint (optional). == Suggestions

  • Use XDR format – for example, use hemelb::io::writers::xdr::XdrMemWriter

  • Use a collective write function from MPI-IO to get information into the filesystem, for example:

    MPI_File_Write_Ordered(fh, bufferContainingXdrDataForLocalParticles, numberOfByteOfXdrDataInBuffer, MPI_CHAR, MPI_STATUS_IGNORE);

  • Output the following fields as a file header:

  • uint32 HemelbMagicNumber (hlb! in ASCII)

  • uint32 ColloidsMagicNumber (col in ASCII)

  • uint32 ColloidVersionNumber (1)

  • Output the following fields as a header:

  • uint32 numberOfBytesOfXdrForHeader

  • uint32 numberOfBytesOfXdrPerParticle

  • uint64 numberOfParticlesInSimulation

  • LatticeTime timestep

  • Output the following fields per particle:

  • uint64 particleId

  • LatticeDistance smallRadius_a0

  • LatticeDistance largeRadius_ah

  • LatticePosition globalPosition

  • LatticeVelocity velocity == Plan SimulationMaster calls ColloidController::OutputInformation every (e.g.) 500 timesteps.

ColloidController::OutputInformation delegates to ParticleSet::OutputInformation.

ParticleSet::OutputInformation

  • (all ranks) use XdrMemWriter to write local particles into xdrBuffer (call Particle::WriteToStream)

  • (all ranks) open output file (append mode)

  • (all ranks) force sync so that file pointer is correct on all ranks: MPI_File_Sync;MPI_Barrier;MPI_File_Sync;

  • (all ranks) remember the absolute displacement in the file: MPI_File_get_position_shared(&offset); MPI_File_get_byte_offset(offset, &startOfHeader)

  • (all ranks) set file view disp=numberOfBytesOfXdrForHeader, etype=MPI_CHAR, filetype=MPI_CHAR

  • (all ranks) force sync so that view is set on all ranks before write begins: MPI_File_Sync;MPI_Barrier;MPI_File_Sync;

  • (all ranks) call MPI_File_Write_Ordered(xdrBuffer, numberOfXdrBytesInBuffer)

  • (all ranks) force sync so that file is correct size: MPI_File_Sync;MPI_Barrier;MPI_File_Sync;

  • (all ranks) remember the relative offset in the file: MPI_File_get_position_shared(&offsetEOF)

  • (all ranks) set file view disp=startOfHeader type=MPI_CHAR, filetype=MPI_CHAR

  • (all ranks) force sync so that view is set on all ranks before write begins: MPI_File_Sync;MPI_Barrier;MPI_File_Sync;

  • (rank == 0) use XdrMemWriter to write header information to xdrBuffer

  • (rank == 0) MPI_File_write(xdrBuffer, startOfHeader, numberOfBytesOfXdrForHeader)

  • (all ranks) close output file == Other requirements Meta-data stored in io::formats::colloids.h

    static const uint16 numberOfBytesOfXdrForHeader = 4+4+8+sizeof(LatticeTime) = 24; static const uint16 numberOfBytesOfXdrPerParticle = 7*8 = 56; == Performance and scaling

  • each rank only writes locally owned particles

  • no global communication except collective IO and "sync;barrier;sync" to ensure data is written to filesystem

  • single episode of collective IO for all particle data enables maximum optimisation in MPI-IO == Extensibility

  • new fields can be added to header in future; reader uses first two bytes to determine where particle data starts in the file and may choose to interpret only some of the header bytes.

  • new fields can be added to particle data; reader uses field in header to determine the length of each particle record in the file and may choose to interpret only some of the particle data bytes.

  • output from multiple timesteps can be stored in the same output file

Clone this wiki locally