University of California, Irvine (UCI) Bag-of-Words format.
http://archive.ics.uci.edu/ml/datasets/Bag+of+Words
Corpus in the UCI bag-of-words format.
Utility method to generate gensim-style Dictionary directly from the corpus and vocabulary data.
Return document at file offset offset (in bytes)
Load a previously saved object from file (also see save).
If the object was saved with large arrays stored separately, you can load these arrays via mmap (shared memory) using mmap=’r’. Default: don’t use mmap, load large arrays as normal objects.
Save a corpus in the UCI Bag-of-Words format.
There are actually two files saved: fname and fname.vocab, where fname.vocab is the vocabulary file.
This function is automatically called by UciCorpus.serialize; don’t call it directly, call serialize instead.
Iterate through the document stream corpus, saving the documents to fname and recording byte offset of each document. Save the resulting index structure to file index_fname (or fname.index is not set).
This relies on the underlying corpus class serializer providing (in addition to standard iteration):
each saved document,
the docbyoffset(offset) method, which returns a document positioned at offset bytes within the persistent storage (file).
Example:
>>> MmCorpus.serialize('test.mm', corpus)
>>> mm = MmCorpus('test.mm') # `mm` document stream now has random access
>>> print(mm[42]) # retrieve document no. 42, etc.
Store a corpus in UCI Bag-of-Words format.
This corpus format is identical to MM format, except for different file headers. There is no format line, and the first three lines of the file contain number_docs, num_terms, and num_nnz, one value per line.
This implementation is based on matutils.MmWriter, and works the same way.
Update headers with actual values.
Write blank header lines. Will be updated later, once corpus stats are known.
Write a single sparse vector to the file.
Sparse vector is any iterable yielding (field id, field value) pairs.