One of the things I do in my day job is develop and maintain C++ firmware for ESP32-based devices. This may be the first post in a series about interesting concepts I encounter during my work on these projects (if I keep it up).
Recently, I was developing a logging system: log specific events from the application to a file, then later process that file and upload it to an API for remote analytics. Events are stored on disk as JSON, but to mitigate corruption1 they're serialized to the log as single objects, one per line. When log files are rotated, each line is deserialized and appended to a JSON array, which is then written out to the "rotated" file:
// active file {"message": "In the midst of chaos"} {"message": "there is also opportunity"} // rotated file [ {"message": "In the midst of chaos"}, {"message": "there is also opportunity"} ] Simple, right?
The Problem
My question is: during rotation, how does one iterate over the lines in the file (to read them as JSON objects)? The constraints are:
- Because of reasons2, we use the C
FILE*API for file I/O, with an RAII wrapper class to provide nicer C++ access to data. - While I could probably fit the entire file in memory, where's the fun in that? It's also not great practice, so instead we should try to lazily read and interpret lines from the file.3
Our interface (and example usage) should look something like:
class file_t { public: file_t(std::filesystem::path path, const char* mode); /* string iterator type */ lines(); private: FILE* m_file; } int main() { file_t file{"art_of_war.txt", "r"}; for (auto& line : file.lines()) { // do something with line, depending on the type } } file_t already exists; we just need to add the lines() method. Without coroutines4, we have a couple of options:
- Read all the lines into a
std::vector<std::string>(but we already decided we don't want to do this). - Write an iterator adapter type. While this would work, it's frustrating: it adds another 1-2 types, state has to be dealt with, iterators are (in my experience) annoying to write, and satisfying the various concepts to make it range-adaptable (a very useful goal) can be fiddly.
A Brief Sidebar to Extol the Virtues of Ranges
Something I've done in other situations is written a function that returns an entire type-deduced range adapter. For example, if we have a file logger class that logs and rotates in a specified folder, we may want to iterate through all of the files that have been rotated:
class file_logger_t { public: file_logger_t(std::filesystem::path directory): m_dir{directory} {} auto logs() const { return std::filesystem::directory_iterator(m_dir) | std::views::filter( [](auto& entry) { return entry.path().extension() == ".json"; }); } private: std::filesystem::path m_dir; } This can then be used like for(const auto& entry : logger.logs()). Ranges are a nice (although often fairly verbose) way to do declarative, lazy-evaluated iteration. Unfortunately this technique can't be applied directly, as we don't have a built-in way to convert a FILE* into a range of lines (being the entire point of this post).
Back to Coroutines
As someone who also uses Python frequently, the obvious thing to use here is a generator:
def read_file(path): with open(path, 'r') as file: for line in file: yield line "But alas," I thought, "I can't do this in C++". Fortunately, I was wrong, and happened to stumble across std::generator, and more generally C++ coroutines.
She co on my routine till I yield
Allow me to give you a brief overview of coroutines, in case you haven't encountered them before. In slightly more formal/traditional/old computing, a function is referred to as a "subroutine": a section of code ("routine") that runs inside another ("sub"). Execution enters the subroutine, some stuff happens, then execution leaves, possibly with a return value.
Coroutines, on the other hand, maintain an internal state:
- The caller invokes the coroutine.
- The coroutine can, at any point, yield, returning execution to the caller, potentially with a value. Critically, all variables are saved.
- The caller does some other stuff.
- The caller returns control to the coroutine.
- The coroutine then resumes from the point it yielded earlier, with all variables containing the same value as was saved earlier.
- This cycle repeats from 2.
- Eventually, the coroutine may return, similarly to a normal function. At this point the variables are discarded.
Coroutines are often referred to as "functions which can be paused and resumed". (They can sort of be approximated with static variables inside functions, but static variables have global state across all invocations, and can't be easily reset).
Feel free to get the point anytime
The upshot of this is the C++ implementation is nearly as simple as the Python one:
class file_t { public: file_t(std::filesystem::path path, const char* mode); std::generator<std::string_view> lines() { char line[4096]; fseek(m_file, 0, SEEK_SET); while (fgets(line, sizeof(line), m_file)) { co_yield std::string_view{line}; } } private: FILE* m_file; } (This doesn't have to implemented in a header file and inline within the class; I've just done so to make it easier).
- I've used
fgets()to read until a newline. This implementation assumes all lines are less than 4096 characters long. - A
std::string_viewdoesn't copy the string fromline, so it's faster thanstd::string(it's effectively returningline*with a nice wrapper). -
fgets()returnsNULLon EOF or error. This implementation also assumes we don't encounter an error.
Disclaimers
- I am not nor am I claiming to be a C++ expert.
- Are there other ways to solve this problem? Certainly! This is just the one I chose to use.
Feel free to discuss, critique my code, ask questions, etc :)
1. Hopefully.
2. #include <iostream> adds about 200 kB to the binary size.
3. Mind you we are storing the entire JSON array in memory, so this is kind of a moot point, but we're sticking with it for the sake of the argument.
4. Because I hadn't heard of them yet
![Quadruped Robot Software — Ch.10: Quadruped AI and Physical AI [KR]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffiibwtdu8wij6gnnuoyc.png)

