Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 34 additions & 20 deletions db.go
Original file line number Diff line number Diff line change
Expand Up @@ -36,12 +36,6 @@ const (
// All data access is performed through transactions which can be obtained through the DB.
// All the functions on DB will return a ErrDatabaseNotOpen if accessed before Open() is called.
type DB struct {
// Put `stats` at the first field to ensure it's 64-bit aligned. Note that
// the first word in an allocated struct can be relied upon to be 64-bit
// aligned. Refer to https://pkg.go.dev/sync/atomic#pkg-note-BUG. Also
// refer to discussion in https://github.com/etcd-io/bbolt/issues/577.
stats Stats
Comment on lines -39 to -43
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry to say this, but did you read the comment? :)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry to say this, but did you read the comment? :)

It changes it to a pointer. So I think it should be fine.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you guys have a raspi or any other arm32 device sitting around to validate this? I'm not so worried about the stats pointer, but rather the sync.Once or sync.Pool could become problematic.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just quickly ran the tests on your branch on my pi - so everything works as intended. Sorry I'm in the middle of a move, luckily I had it plugged in and could ssh into it. Not sure whether you have your QEMU setup still somewhere...

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you guys have a raspi or any other arm32 device sitting around to validate this?

qemu-user-static might be good enough to verify this. #577 (comment)

I think we might need to setup a workflow to guarantee it won't be broken in future.

but rather the sync.Once or sync.Pool could become problematic.

The known issue https://pkg.go.dev/sync/atomic#pkg-note-BUG seems only specific to sync.atomic package. But I agree that we should verify it anyway.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we might need to setup a workflow to guarantee it won't be broken in future.

cc @ivanvc

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we might need to setup a workflow to guarantee it won't be broken in future.

On it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx @ivanvc pls refer to #665


// When enabled, the database will perform a Check() after every commit.
// A panic is issued if the database is in an inconsistent state. This
// flag has a large performance impact so it should only be used for
Expand Down Expand Up @@ -138,6 +132,7 @@ type DB struct {
pageSize int
opened bool
rwtx *Tx
stats *Stats

freelist fl.Interface
freelistLoad sync.Once
Expand Down Expand Up @@ -203,6 +198,10 @@ func Open(path string, mode os.FileMode, options *Options) (db *DB, err error) {
db.MaxBatchDelay = common.DefaultMaxBatchDelay
db.AllocSize = common.DefaultAllocSize

if !options.NoStatistics {
db.stats = new(Stats)
}

if options.Logger == nil {
db.logger = getDiscardLogger()
} else {
Expand Down Expand Up @@ -430,7 +429,9 @@ func (db *DB) loadFreelist() {
// Read free list from freelist page.
db.freelist.Read(db.page(db.meta().Freelist()))
}
db.stats.FreePageN = db.freelist.FreeCount()
if db.stats != nil {
db.stats.FreePageN = db.freelist.FreeCount()
}
})
}

Expand Down Expand Up @@ -808,10 +809,12 @@ func (db *DB) beginTx() (*Tx, error) {
db.metalock.Unlock()

// Update the transaction stats.
db.statlock.Lock()
db.stats.TxN++
db.stats.OpenTxN++
db.statlock.Unlock()
if db.stats != nil {
db.statlock.Lock()
db.stats.TxN++
db.stats.OpenTxN++
db.statlock.Unlock()
}

return t, nil
}
Expand Down Expand Up @@ -867,10 +870,12 @@ func (db *DB) removeTx(tx *Tx) {
db.metalock.Unlock()

// Merge statistics.
db.statlock.Lock()
db.stats.OpenTxN--
db.stats.TxStats.add(&tx.stats)
db.statlock.Unlock()
if db.stats != nil {
db.statlock.Lock()
db.stats.OpenTxN--
db.stats.TxStats.add(&tx.stats)
db.statlock.Unlock()
}
}

// Update executes a function within the context of a read-write managed transaction.
Expand Down Expand Up @@ -1088,9 +1093,13 @@ func (db *DB) Sync() (err error) {
// Stats retrieves ongoing performance stats for the database.
// This is only updated when a transaction closes.
func (db *DB) Stats() Stats {
db.statlock.RLock()
defer db.statlock.RUnlock()
return db.stats
var s Stats
if db.stats != nil {
db.statlock.RLock()
s = *db.stats
db.statlock.RUnlock()
}
return s
}

// This is for internal access to the raw data bytes from the C cursor, use
Expand Down Expand Up @@ -1340,15 +1349,20 @@ type Options struct {

// Logger is the logger used for bbolt.
Logger Logger

// NoStatistics turns off statistics collection, Stats method will
// return empty structure in this case. This can be beneficial for
// performance under high-concurrency read-only transactions.
NoStatistics bool
}

func (o *Options) String() string {
if o == nil {
return "{}"
}

return fmt.Sprintf("{Timeout: %s, NoGrowSync: %t, NoFreelistSync: %t, PreLoadFreelist: %t, FreelistType: %s, ReadOnly: %t, MmapFlags: %x, InitialMmapSize: %d, PageSize: %d, MaxSize: %d, NoSync: %t, OpenFile: %p, Mlock: %t, Logger: %p}",
o.Timeout, o.NoGrowSync, o.NoFreelistSync, o.PreLoadFreelist, o.FreelistType, o.ReadOnly, o.MmapFlags, o.InitialMmapSize, o.PageSize, o.MaxSize, o.NoSync, o.OpenFile, o.Mlock, o.Logger)
return fmt.Sprintf("{Timeout: %s, NoGrowSync: %t, NoFreelistSync: %t, PreLoadFreelist: %t, FreelistType: %s, ReadOnly: %t, MmapFlags: %x, InitialMmapSize: %d, PageSize: %d, MaxSize: %d, NoSync: %t, OpenFile: %p, Mlock: %t, Logger: %p, NoStatistics: %t}",
o.Timeout, o.NoGrowSync, o.NoFreelistSync, o.PreLoadFreelist, o.FreelistType, o.ReadOnly, o.MmapFlags, o.InitialMmapSize, o.PageSize, o.MaxSize, o.NoSync, o.OpenFile, o.Mlock, o.Logger, o.NoStatistics)

}

Expand Down
16 changes: 9 additions & 7 deletions tx.go
Original file line number Diff line number Diff line change
Expand Up @@ -357,13 +357,15 @@ func (tx *Tx) close() {
tx.db.rwlock.Unlock()

// Merge statistics.
tx.db.statlock.Lock()
tx.db.stats.FreePageN = freelistFreeN
tx.db.stats.PendingPageN = freelistPendingN
tx.db.stats.FreeAlloc = (freelistFreeN + freelistPendingN) * tx.db.pageSize
tx.db.stats.FreelistInuse = freelistAlloc
tx.db.stats.TxStats.add(&tx.stats)
tx.db.statlock.Unlock()
if tx.db.stats != nil {
tx.db.statlock.Lock()
tx.db.stats.FreePageN = freelistFreeN
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking if we can use atomic to update stats here. so no more lock and no more extra option for openning db.
anyway, changes look good.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking if we can use atomic to update stats here

I think using atomic functions is problematic, since it only guarantees atomic of one operation, but it doesn't guarantee atomic of multiple operations.

tx.db.stats.PendingPageN = freelistPendingN
tx.db.stats.FreeAlloc = (freelistFreeN + freelistPendingN) * tx.db.pageSize
tx.db.stats.FreelistInuse = freelistAlloc
tx.db.stats.TxStats.add(&tx.stats)
tx.db.statlock.Unlock()
}
} else {
tx.db.removeTx(tx)
}
Expand Down