Package journal
import "github.com/syndtr/goleveldb/leveldb/journal"
- Overview
- Index
Package journal reads and writes sequences of journals. Each journal is a stream
of bytes that completes before the next journal starts.
When reading, call Next to obtain an io.Reader for the next journal. Next will
return io.EOF when there are no more journals. It is valid to call Next
without reading the current journal to exhaustion.
When writing, call Next to obtain an io.Writer for the next journal. Calling
Next finishes the current journal. Call Close to finish the final journal.
Optionally, call Flush to finish the current journal and flush the underlying
writer without starting a new journal. To start a new journal after flushing,
call Next.
Neither Readers or Writers are safe to use concurrently.
Example code:
func read(r io.Reader) ([]string, error) {
var ss []string
journals := journal.NewReader(r, nil, true, true)
for {
j, err := journals.Next()
if err == io.EOF {
break
}
if err != nil {
return nil, err
}
s, err := ioutil.ReadAll(j)
if err != nil {
return nil, err
}
ss = append(ss, string(s))
}
return ss, nil
}
func write(w io.Writer, ss []string) error {
journals := journal.NewWriter(w)
for _, s := range ss {
j, err := journals.Next()
if err != nil {
return err
}
if _, err := j.Write([]byte(s)), err != nil {
return err
}
}
return journals.Close()
}
The wire format is that the stream is divided into 32KiB blocks, and each
block contains a number of tightly packed chunks. Chunks cannot cross block
boundaries. The last block may be shorter than 32 KiB. Any unused bytes in a
block must be zero.
A journal maps to one or more chunks. Each chunk has a 7 byte header (a 4
byte checksum, a 2 byte little-endian uint16 length, and a 1 byte chunk type)
followed by a payload. The checksum is over the chunk type and the payload.
There are four chunk types: whether the chunk is the full journal, or the
first, middle or last chunk of a multi-chunk journal. A multi-chunk journal
has one first chunk, zero or more middle chunks, and one last chunk.
The wire format allows for limited recovery in the face of data corruption:
on a format error (such as a checksum mismatch), the reader moves to the
next block and looks for the next full or first chunk.
In the call graph viewer below, each node
is a function belonging to this package
and its children are the functions it
calls—perhaps dynamically.
The root nodes are the entry points of the
package: functions that may be called from
outside the package.
There may be non-exported or anonymous
functions among them if they are called
dynamically from another package.
Click a node to visit that function's source code.
From there you can visit its callers by
clicking its declaring func
token.
Functions may be omitted if they were
determined to be unreachable in the
particular programs or tests that were
analyzed.
type Dropper interface {
Drop(err error)
}
Dropper is the interface that wrap simple Drop method. The Drop
method will be called when the journal reader dropping a block or chunk.
type ErrCorrupted struct {
Size int
Reason string
}
ErrCorrupted is the error type that generated by corrupted block or chunk.
func (*ErrCorrupted) Error
func (e *ErrCorrupted) Error() string
type Reader struct {
}
Reader reads journals from an underlying io.Reader.
func NewReader(r io.Reader, dropper Dropper, strict, checksum bool) *Reader
NewReader returns a new reader. The dropper may be nil, and if
strict is true then corrupted or invalid chunk will halt the journal
reader entirely.
func (*Reader) Next
func (r *Reader) Next() (io.Reader, error)
Next returns a reader for the next journal. It returns io.EOF if there are no
more journals. The reader returned becomes stale after the next Next call,
and should no longer be used. If strict is false, the reader will returns
io.ErrUnexpectedEOF error when found corrupted journal.
func (*Reader) Reset
func (r *Reader) Reset(reader io.Reader, dropper Dropper, strict, checksum bool) error
Reset resets the journal reader, allows reuse of the journal reader. Reset returns
last accumulated error.
type Writer struct {
}
Writer writes journals to an underlying io.Writer.
func NewWriter(w io.Writer) *Writer
NewWriter returns a new Writer.
func (*Writer) Close
func (w *Writer) Close() error
Close finishes the current journal and closes the writer.
func (*Writer) Flush
func (w *Writer) Flush() error
Flush finishes the current journal, writes to the underlying writer, and
flushes it if that writer implements interface{ Flush() error }.
func (*Writer) Next
func (w *Writer) Next() (io.Writer, error)
Next returns a writer for the next journal. The writer returned becomes stale
after the next Close, Flush or Next call, and should no longer be used.
func (*Writer) Reset
func (w *Writer) Reset(writer io.Writer) (err error)
Reset resets the journal writer, allows reuse of the journal writer. Reset
will also closes the journal writer if not already.