libboost-tokenizer/1.77.0+1

[brief]

Break of a string or other character sequence into a series of tokens

Boost.Tokenizer is a part of Boost C++ Libraries. The Boost.Tokenizer package provides a flexible and easy-to-use way to break a string or other character sequence into a series of tokens.

License

Distributed under the Boost Software License, Version 1.0.

Properties

Build Status

Branch Travis Appveyor Coverity Scan codecov.io Deps Docs Tests
master Build Status Build status Coverity Scan Build Status codecov Deps Documentation Enter the Matrix
develop Build Status Build status Coverity Scan Build Status codecov Deps Documentation Enter the Matrix

Overview

break up a phrase into words.

Try it online

#include <iostream>
#include <boost/tokenizer.hpp>
#include <string>

int main(){
    std::string s = "This is,  a test";
    typedef boost::tokenizer<> Tok;
    Tok tok(s);
    for (Tok::iterator beg = tok.begin(); beg != tok.end(); ++beg){
        std::cout << *beg << "\n";
    }
}

Using Range-based for loop (>c++11)

Try it online

#include <iostream>
#include <boost/tokenizer.hpp>
#include <string>

int main(){
    std::string s = "This is,  a test";
    boost::tokenizer<> tok(s);
    for (auto token: tok) {
        std::cout << token << "\n";
    }
}

Documentation

Documentation can be found at Boost.Tokenizer

Related Material

Boost.Tokenizer Chapter 10 at theboostcpplibraries.com, contains several examples including escaped_list_separator.

##Contributing

This library is being maintained as a part of the Boost Library Official Maintainer Program

Open Issues on

##Acknowledgements

From the author:

I wish to thank the members of the boost mailing list, whose comments, compliments, and criticisms during both the development and formal review helped make the Tokenizer library what it is. I especially wish to thank Aleksey Gurtovoy for the idea of using a pair of iterators to specify the input, instead of a string. I also wish to thank Jeremy Siek for his idea of providing a container interface for the token iterators and for simplifying the template parameters for the TokenizerFunctions. He and Daryle Walker also emphasized the need to separate interface and implementation. Gary Powell sparked the idea of using the isspace and ispunct as the defaults for char_delimiters_separator. Jeff Garland provided ideas on how to change to order of the template parameters in order to make tokenizer easier to declare. Thanks to Douglas Gregor who served as review manager and provided many insights both on the boost list and in e-mail on how to polish up the implementation and presentation of Tokenizer. Finally, thanks to Beman Dawes who integrated the final version into the boost distribution.

##License Distributed under the Boost Software License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)

version 1.77.0+1
license BSL-1.0Boost Software License 1.0
repository https://pkg.cppget.org/1/stable
download libboost-tokenizer-1.77.0+1.tar.gz
sha256 cd3c28be9abae537fc15ba52cfa56ecb6bb65b692dbe65b62677e2faee3081a5
project boost
url github.com/boostorg/tokenizer
doc-url www.boost.org/doc/libs/1_77_0/libs/tokenizer
package-url github.com/build2-packaging/boost
package-email packaging@build2.orgMailing list
topics C++Boost

Depends (6)

libboost-assert == 1.77.0
libboost-config == 1.77.0
libboost-iterator == 1.77.0
libboost-mpl == 1.77.0
libboost-throw-exception == 1.77.0
libboost-type-traits == 1.77.0

Builds

toolchain public-0.16.0
target aarch64-linux-gnu
tgt config linux_debian_12-gcc_13-static_O3
timestamp 2024-03-29 05:48:42 UTC (06:39:40 hours ago)
result success | log | rebuild
toolchain public-0.16.0
target aarch64-linux-gnu
tgt config linux_debian_12-gcc_13-ndebug_O3
timestamp 2024-03-29 05:47:56 UTC (06:40:26 hours ago)
result success | log | rebuild
toolchain public-0.16.0
target aarch64-linux-gnu
tgt config linux_debian_12-gcc_13-O3
timestamp 2024-03-29 05:45:57 UTC (06:42:25 hours ago)
result success | log | rebuild
toolchain public-0.16.0
target aarch64-linux-gnu
tgt config linux_debian_12-gcc_13
timestamp 2024-03-29 05:45:03 UTC (06:43:20 hours ago)
result success | log | rebuild
toolchain public-0.16.0
target aarch64-linux-gnu
tgt config linux_debian_12-clang_16_libc++-static_O3
timestamp 2024-03-29 00:00:01 UTC (12:28:21 hours ago)
result success | log | rebuild
toolchain public-0.16.0
target aarch64-linux-gnu
tgt config linux_debian_12-clang_16_libc++-O3
timestamp 2024-03-28 23:49:16 UTC (12:39:07 hours ago)
result success | log | rebuild
toolchain public-0.16.0
target aarch64-linux-gnu
tgt config linux_debian_12-clang_16_libc++
timestamp 2024-03-28 23:38:54 UTC (12:49:28 hours ago)
result success | log | rebuild
toolchain public-0.16.0
target aarch64-linux-gnu
tgt config linux_debian_12-clang_16-O3
timestamp 2024-03-28 23:32:22 UTC (12:56:00 hours ago)
result success | log | rebuild
toolchain public-0.16.0
target aarch64-linux-gnu
tgt config linux_debian_12-clang_16-static_O3
timestamp 2024-03-28 23:25:47 UTC (13:02:35 hours ago)
result success | log | rebuild
toolchain public-0.16.0
target aarch64-linux-gnu
tgt config linux_debian_12-clang_16
timestamp 2024-03-28 23:18:25 UTC (13:09:57 hours ago)
result success | log | rebuild
toolchain public-0.16.0
target aarch64-linux-gnu
tgt config linux_debian_12-clang_15_libc++
timestamp 2024-03-28 14:27:24 UTC (22:00:58 hours ago)
result success | log | rebuild
toolchain public-0.16.0
target aarch64-linux-gnu
tgt config linux_debian_12-clang_15
timestamp 2024-03-28 13:57:22 UTC (22:31:00 hours ago)
result success | log | rebuild
toolchain public-0.16.0
target aarch64-linux-gnu
tgt config linux_debian_11-gcc_12
timestamp 2024-03-28 13:17:31 UTC (23:10:51 hours ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-apple-darwin22.5.0
tgt config macos_13-clang_15.0-static_O3
timestamp 2024-03-28 09:02:20 UTC (01 03:26:02 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-apple-darwin22.5.0
tgt config macos_13-clang_15.0-O3
timestamp 2024-03-28 09:01:08 UTC (01 03:27:15 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-apple-darwin22.5.0
tgt config macos_13-clang_15.0
timestamp 2024-03-28 08:56:52 UTC (01 03:31:30 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-linux-gnu
tgt config linux_fedora_38-gcc_13-bindist
timestamp 2024-03-28 04:22:36 UTC (01 08:05:46 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-linux-gnu
tgt config linux_fedora_37-gcc_12.2-bindist
timestamp 2024-03-28 01:10:09 UTC (01 11:18:13 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-linux-gnu
tgt config linux_debian_12-gcc_13.1-static_O3
timestamp 2024-03-27 18:47:03 UTC (01 17:41:19 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-linux-gnu
tgt config linux_debian_12-gcc_13.1-ndebug_O3
timestamp 2024-03-27 18:45:53 UTC (01 17:42:29 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-linux-gnu
tgt config linux_debian_12-gcc_13.1-O3
timestamp 2024-03-27 18:45:08 UTC (01 17:43:14 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-linux-gnu
tgt config linux_debian_12-gcc_13.1
timestamp 2024-03-27 18:44:35 UTC (01 17:43:47 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-microsoft-win32-msvc14.3
tgt config windows_10-msvc_17.6-O2
timestamp 2024-03-27 18:26:03 UTC (01 18:02:19 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-microsoft-win32-msvc14.3
tgt config windows_10-msvc_17.6-static_O2
timestamp 2024-03-27 18:25:08 UTC (01 18:03:14 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-microsoft-win32-msvc14.3
tgt config windows_10-msvc_17.6
timestamp 2024-03-27 18:22:30 UTC (01 18:05:52 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-w64-mingw32
tgt config windows_10-gcc_12.2_mingw_w64
timestamp 2024-03-27 18:21:42 UTC (01 18:06:40 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-w64-mingw32
tgt config windows_10-gcc_12.2_mingw_w64-static_O2
timestamp 2024-03-27 18:19:02 UTC (01 18:09:20 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-w64-mingw32
tgt config windows_10-gcc_12.2_mingw_w64-O2
timestamp 2024-03-27 18:18:28 UTC (01 18:09:54 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-linux-gnu
tgt config linux_debian_12-clang_16.0-static_O3
timestamp 2024-03-27 18:07:19 UTC (01 18:21:03 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-linux-gnu
tgt config linux_debian_12-clang_16.0_libc++-static_O3
timestamp 2024-03-27 18:06:41 UTC (01 18:21:41 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-linux-gnu
tgt config linux_debian_12-clang_16.0_libc++
timestamp 2024-03-27 18:06:36 UTC (01 18:21:46 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-linux-gnu
tgt config linux_debian_12-clang_16.0
timestamp 2024-03-27 18:05:49 UTC (01 18:22:33 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-linux-gnu
tgt config linux_debian_12-clang_16.0-O3
timestamp 2024-03-27 18:05:25 UTC (01 18:22:57 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-linux-gnu
tgt config linux_debian_12-gcc_12-bindist
timestamp 2024-03-27 18:04:52 UTC (01 18:23:30 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-linux-gnu
tgt config linux_debian_12-clang_16.0_libc++-O3
timestamp 2024-03-27 18:04:11 UTC (01 18:24:11 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-microsoft-win32-msvc14.3
tgt config windows_10-clang_16.0_llvm_msvc_17.6-static_O2
timestamp 2024-03-27 17:50:54 UTC (01 18:37:29 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-microsoft-win32-msvc14.3
tgt config windows_10-clang_16.0_llvm_msvc_17.6-O2
timestamp 2024-03-27 17:49:51 UTC (01 18:38:31 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-microsoft-win32-msvc14.3
tgt config windows_10-clang_16.0_llvm_msvc_17.6
timestamp 2024-03-27 17:47:32 UTC (01 18:40:50 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-microsoft-win32-msvc14.3
tgt config windows_10-clang_15.0_msvc_msvc_17.6
timestamp 2024-03-27 16:12:32 UTC (01 20:15:50 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-microsoft-win32-msvc14.3
tgt config windows_10-msvc_17.5
timestamp 2024-03-27 16:11:26 UTC (01 20:16:56 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-freebsd13.2
tgt config freebsd_13-clang_14.0-static_O3
timestamp 2024-03-27 16:02:18 UTC (01 20:26:04 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-freebsd13.2
tgt config freebsd_13-clang_14.0
timestamp 2024-03-27 16:00:24 UTC (01 20:27:58 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-freebsd13.2
tgt config freebsd_13-clang_14.0-O3
timestamp 2024-03-27 16:00:06 UTC (01 20:28:16 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-linux-gnu
tgt config linux_debian_12-clang_15.0_libc++
timestamp 2024-03-27 15:44:17 UTC (01 20:44:05 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-freebsd12.4
tgt config freebsd_12-clang_13.0
timestamp 2024-03-27 15:43:10 UTC (01 20:45:12 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-linux-gnu
tgt config linux_debian_12-clang_15.0
timestamp 2024-03-27 15:43:01 UTC (01 20:45:21 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-linux-gnu
tgt config linux_debian_11-gcc_12.1
timestamp 2024-03-27 15:41:21 UTC (01 20:47:01 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-apple-darwin22.5.0
tgt config macos_13-clang_14.0-static_O3
timestamp 2024-03-27 01:45:48 UTC (02 10:42:34 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-apple-darwin22.5.0
tgt config macos_13-clang_14.0-O3
timestamp 2024-03-27 01:39:49 UTC (02 10:48:33 days ago)
result success | log | rebuild
toolchain public-0.16.0
target x86_64-apple-darwin22.5.0
tgt config macos_13-clang_14.0
timestamp 2024-03-27 01:37:36 UTC (02 10:50:47 days ago)
result success | log | rebuild
target x86_64-apple-darwin22.5.0
tgt config macos_13-gcc_13_homebrew
result excluded (https://github)
target x86_64-apple-darwin22.5.0
tgt config macos_13-gcc_13_homebrew-O3
result excluded (https://github)
target x86_64-apple-darwin22.5.0
tgt config macos_13-gcc_13_homebrew-static_O3
result excluded (https://github)