Thu, 06 Feb 2025 20:09:55 +0100
write introduction for the Utilities Section
relates to #451
1145
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
1 | # Data Streams |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
2 | |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
3 | Stream copy functions provide a way to copy all - or a limited amount of - data from one stream to another. |
1166
03bbdf402675
write introduction for the Utilities Section
Mike Becker <universe@uap-core.de>
parents:
1147
diff
changeset
|
4 | |
1145
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
5 | Since the read/write functions of a [UCX buffer](buffer.h.md) are fully compatible with stream read/write functions, |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
6 | you can, for example, easily transfer data from a file or network stream to a UCX buffer or vice versa. |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
7 | |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
8 | ## Overview |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
9 | ```C |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
10 | #include <cx/streams.h> |
1141 | 11 | |
1145
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
12 | size_t cx_stream_copy( |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
13 | void *src, void *dest, |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
14 | cx_read_func rfnc, cx_write_func wfnc |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
15 | ); |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
16 | size_t cx_stream_ncopy( |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
17 | void *src, void *dest, |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
18 | cx_read_func rfnc, cx_write_func wfnc, |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
19 | size_t n |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
20 | ); |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
21 | |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
22 | size_t cx_stream_bcopy( |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
23 | void *src, void *dest, |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
24 | cx_read_func rfnc, cx_write_func wfnc, |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
25 | char *buf, size_t bufsize |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
26 | ); |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
27 | size_t cx_stream_bncopy( |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
28 | void *src, void *dest, |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
29 | cx_read_func rfnc, cx_write_func wfnc, |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
30 | char *buf, size_t bufsize, |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
31 | size_t n |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
32 | ); |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
33 | ``` |
1141 | 34 | |
1145
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
35 | ## Description |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
36 | |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
37 | All functions in the stream copy family use the `rfnc` to read data from `src` |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
38 | and use the `wfnc` to write the data to `dest`. |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
39 | |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
40 | The `cx_stream_copy()` function always uses internal stack memory as a temporary buffer for the read bytes. |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
41 | The `cx_stream_bcopy()` function uses either a pre-initialized buffer `buf` of length `bufsize` |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
42 | or, if `buf` is `NULL`, an internal heap-allocated buffer. |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
43 | |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
44 | The `cx_stream_ncopy()` function behaves like `cx_stream_copy()` except, that it reads at most `n` bytes |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
45 | (and the same is true for `cx_stream_bncopy()` and `cx_stream_bcopy()`). |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
46 | |
1141 | 47 | |
1145
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
48 | <warning> |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
49 | When you are reading from a stream where you cannot track the position, there is the possibility that |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
50 | data gets lost when the destination does not accept all the bytes read from the source. |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
51 | While the stream copy functions do report how many bytes were <emphasis>successfully</emphasis> copied |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
52 | to the destination, this might - in certain cases - not be the exact number of read items. |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
53 | |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
54 | To mitigate the risk, you should make sure that the destination can always accept all read bytes and |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
55 | a possible bottleneck is only introduced by the source. |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
56 | </warning> |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
57 | |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
58 | > The size of the internal _stack_ buffer in `cx_stream_copy()` and `cx_stream_ncopy()` can be |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
59 | > set during compilation via the `CX_STREAM_COPY_BUF_SIZE` macro. |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
60 | > Similarly, the size for the implicitly allocated _heap_ buffer in can be |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
61 | > configured via the `CX_STREAM_BCOPY_BUF_SIZE` macro. |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
62 | > Refer to the [build instructions](install.md#compile-time-options) for the details. |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
63 | |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
64 | ## Example |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
65 | |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
66 | The following example shows, how to read the contents of a file into a buffer: |
1141 | 67 | ```c |
68 | FILE *inputfile = fopen(infilename, "r"); | |
69 | if (inputfile) { | |
70 | CxBuffer fbuf; | |
71 | cxBufferInit(&fbuf, NULL, 4096, NULL, CX_BUFFER_AUTO_EXTEND); | |
72 | cx_stream_copy(inputfile, &fbuf, | |
73 | (cx_read_func) fread, | |
74 | cxBufferWriteFunc); | |
75 | fclose(inputfile); | |
76 | ||
77 | // ... do something meaningful with the contents ... | |
78 | ||
79 | cxBufferDestroy(&fbuf); | |
80 | } else { | |
81 | perror("Error opening input file"); | |
82 | } | |
83 | ``` | |
1142
9437530176bc
add symbols that need documentation as TODOs
Mike Becker <universe@uap-core.de>
parents:
1141
diff
changeset
|
84 | |
1145
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
85 | |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
86 | <seealso> |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
87 | <category ref="apidoc"> |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
88 | <a href="https://ucx.sourceforge.io/api/streams_8h.html">streams.h</a> |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
89 | </category> |
1a8fe7b7dd8a
add stream.h docs and reworks hash_key.h docs
Mike Becker <universe@uap-core.de>
parents:
1143
diff
changeset
|
90 | </seealso> |