By the way I'm using int64 timestamps instead of time.Time.
This isn't actually about time at all, we can drop that. You want to look up integers within a given range.
Rather than trying to make up your own indexing solution, probably your best solution is to put your data into a SQLite temporary database, index, and query it.
create table data ( num biginteger not null, -- add your data columns);create index data_num_idx on data(num);
This can be coded up quickly. Decide on your performance criteria and run some benchmarks while you're coding up a Go solution.
If you want to do this yourself, you can do exactly what SQLite does; use a B-Tree index. Store the timestamp as the key. Search for the node which is equal to the lower bound, or until you fail to match, then walk the tree until you hit a node greater than the upper bound.
There are several Go B-Tree implementations out there.