NERsuite  1.1.1
Public Member Functions
NER::Tokenizer Class Reference

#include <tokenizer.h>

List of all members.

Public Member Functions

int tokenize (const std::string &raw_sent, V2_STR &data, size_t init_offset=0)

Detailed Description

Tokenizer Utility


Member Function Documentation

int NER::Tokenizer::tokenize ( const std::string &  raw_sent,
V2_STR &  data,
size_t  init_offset = 0 
)

Tokenize input text and generate the (begin_pos, end_pos, token) triplet for each token.

Parameters:
[in]raw_sentInput sentence
[in,out]dataA vector to store the result
[in]init_offsetThe offset value added to all begin_pos
Returns:
the count of tokens generated.

The documentation for this class was generated from the following files:
 All Classes Functions Variables