tokenizer/char_delimiters_separator.htm at develop · brbzull0/tokenizer · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

<html>
<head>
  <meta http-equiv="Content-Language" content="en-us">
  <meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
  <meta name="GENERATOR" content="Microsoft FrontPage 6.0">
  <meta name="ProgId" content="FrontPage.Editor.Document">

  <title>Boost Char Delimiters Separator</title>
</head>

<body bgcolor="#FFFFFF" text="#000000" link="#0000EE" vlink="#551A8B" alink=
"#FF0000">
  <p><img src="../../boost.png" alt="C++ Boost" width="277" height=
  "86"><br></p><font color="red">Note: This class is deprecated. Please use
  <a href="char_separator.htm"><tt>char_separator</tt></a> instead.</font>

  <h1 align="center">Char Delimiters Separator</h1>
  <pre>
template &lt;class Char, class Traits = std::char_traits&lt;Char&gt; &gt;
class char_delimiters_separator{
</pre>

  <p>The char_delimiters_separator class is an implementation of the <a href=
  "tokenizerfunction.htm">TokenizerFunction</a> concept that can be used to
  break text up into tokens. It is the default TokenizerFunction for
  tokenizer and token_iterator_generator. An example is below.</p>

  <h2>Example</h2>
  <pre>
// simple_example_4.cpp
#include&lt;iostream&gt;
#include&lt;boost/tokenizer.hpp&gt;
#include&lt;string&gt;

int main(){
   using namespace std;
   using namespace boost;
   string s = "This is,  a test";
   tokenizer&lt;char_delimiters_separator&lt;char&gt; &gt; tok(s);
   for(tokenizer&lt;char_delimiters_separator&lt;char&gt; &gt;::iterator beg=tok.begin(); beg!=tok.end();++beg){
       cout &lt;&lt; *beg &lt;&lt; "\n";
   }
}
</pre>

  <h2>Construction and Usage</h2>

  <p>There is one constructor of interest. It is as follows</p>
  <pre>
explicit char_delimiters_separator(bool return_delims = false,
const Char* returnable = "",const Char* nonreturnable = "" )
</pre>

  <table border="1" summary="">
    <tr>
      <td>
        <p align="center"><strong>Parameter</strong></p>
      </td>

      <td>
        <p align="center"><strong>Description</strong></p>
      </td>
    </tr>

    <tr>
      <td>return_delims</td>

      <td>Whether or not to return the delimiters that have been found. Note
      that not all delimiters can be returned. See the other two parameters
      for explanation.</td>
    </tr>

    <tr>
      <td>returnable</td>

      <td>This specifies the returnable delimiters. These are the delimiters
      that can be returned as tokens when return_delims is true. Since these
      are typically punctuation, if a 0 is provided as the argument, then the
      returnable delmiters will be all characters Cfor which std::ispunct(C)
      yields a true value. If an argument of "" is provided, then this is
      taken to mean that there are noreturnable delimiters.</td>
    </tr>

    <tr>
      <td>nonreturnable</td>

      <td>This specifies the nonreturnable delimiters. These are delimiters
      that cannot be returned as tokens. Since these are typically
      whitespace, if 0 is specified as an argument, then the nonreturnable
      delimiters will be all characters C for which std::isspace(C) yields a
      true value. If an argument of "" is provided, then this is taken to
      mean that there are no non-returnable delimiters.</td>
    </tr>
  </table>

  <p>The reason there is a distinction between nonreturnable and returnable
  delimiters is that some delimiters are just used to split up tokens and are
  nothing more. Take for example the following string "b c +". Assume you are
  writing a simple calculator to parse expression in post fix notation. While
  both the space and the + separate tokens, you only only interested in the +
  and not in the space. Indeed having the space returned as a token would
  only complicate your code. In this case you would specify + as a
  returnable, and space as a nonreturnable delimiter.</p>

  <p>To use this class, pass an object of it anywhere a TokenizerFunction
  object is required.</p>

  <h3>Template Parameters</h3>

  <table border="1" summary="">
    <tr>
      <th>Parameter</th>

      <th>Description</th>
    </tr>

    <tr>
      <td><tt>Char</tt></td>

      <td>The type of the elements within a token, typically
      <tt>char</tt>.</td>
    </tr>

    <tr>
      <td>Traits</td>

      <td>The traits class for Char, typically
      std::char_traits&lt;Char&gt;</td>
    </tr>
  </table>

  <h2>Model of</h2>

  <p><a href="tokenizerfunction.htm">TokenizerFunction</a></p>

  <p>&nbsp;</p>
  <hr>

  <p><a href="http://validator.w3.org/check?uri=referer"><img border="0" src=
  "../../doc/images/valid-html401.png" alt="Valid HTML 4.01 Transitional"
  height="31" width="88"></a></p>

  <p>Revised
  <!--webbot bot="Timestamp" s-type="EDITED" s-format="%d %B, %Y" startspan -->25
  December, 2006<!--webbot bot="Timestamp" endspan i-checksum="38518" --></p>

  <p><i>Copyright &copy; 2001 John R. Bandela</i></p>

  <p><i>Distributed under the Boost Software License, Version 1.0. (See
  accompanying file <a href="../../LICENSE_1_0.txt">LICENSE_1_0.txt</a> or
  copy at <a href=
  "http://www.boost.org/LICENSE_1_0.txt">http://www.boost.org/LICENSE_1_0.txt</a>)</i></p>
</body>
</html>